The brain has a memory, i.e. a unique ability to memorize and reproduce information. Memory mechanisms consist of perception, memorization, storage and subsequent reproduction of information.
Based on numerous studies, only the neurophysiological mechanisms of perception, memorization and storage of information in the brain are disclosed, which include neurochemical and molecular processes. At the same time, the key mechanisms for extracting information from memory are still completely unknown. The problem is that the extraction of information from memory occurs in the subjective, mental sphere of brain activity, which was not available for study by existing neurophysiological methods.
In recent years, progressive wavelet transforms based electroencephalogram (EEG) methods have been developed. Using the mathematical wavelet transform based electroencephalogram (EEG) method, we established the fundamental possibility of objective recording of the mental activity of the human brain, which opened the prospect of revealing the mechanisms for extracting information from memory.
The overall goal of this study is to develop a fundamentally new information technology to identify mechanisms of memory reproduction in the mental activity of the brain by means of the wavelet transform based electroencephalogram (EEG). To solve this problem, an experimental model and software that allow testing the extraction of information from memory in humans have been developed [1-6].
information technology, brain, electroencephalogram (EEG), wavelet transform based EEG, mental activity, memory, information retrieval from memory
The brain is a unique organization in living nature, possessing the ability to mental activity, which manifests itself through consciousness, in thoughts, feelings, emotions, memory, i.e. through the person's subjective perception of oneself and the world around.
The need to study the nature of the mental activity of the brain was indicated by I.P. Pavlov, P.K. Anokhin, N.P. Bekhtereva, A.M. Ivanitsky K.V. Sudakov, T. Nagel, K. Popper, R.W. Sperry, F. Crick, and C. Koch [7-15].
The brain has a memory, i.e. a unique ability to memorize and reproduce information. Memory mechanisms consist of perception, memorization, storage and subsequent reproduction of information. Based on numerous studies, only the neurophysiological mechanisms of perception, memorization and storage of information in the brain are disclosed, which include neurochemical and molecular processes [16-19].
At the same time, the key mechanisms for extracting information from memory are still completely unknown. The problem is that the extraction of information from memory occurs in the subjective, mental sphere of brain activity, which was not available for study by existing neurophysiological methods until recently [20-22].
In recent years, progressive wavelet transforms based electroencephalogram (EEG) methods have been developed [1-4]. Using the wavelet transform based EEG method, we established the fundamental possibility of direct objective recording of the mental activity of the human brain [5,6], which opened the prospect of developing a new information technology for studying the mechanisms of extracting information from the human brain memory.
The overall goal of the work is the development of a fundamentally new information technology for detecting within the mental activity of the brain the extraction of information from memory by means of the wavelet transform based electroencephalogram (EEG).
To solve this problem, it is necessary to develop an experimental model and software that allows testing extraction of information from human memory.
The PC-based experimental setup has two monitors that are simultaneously used during testing of the test person's mental responses.
The first monitor is intended for the experimenter who adjusts settings in the Control Panel main window of the program (Figure 1).
Figure 1. View of the main window on the monitor for the experimenter
The Control panel window consists of several parts:
1. The list of questions and images that will be shown to the test person is displayed in the upper left part.
2. In the upper right part, there is a panel with experiment settings and buttons for controlling the progress of the experiment.
3. In the lower left part, there is the window of the experiment log, where text information about the testing progress is displayed.
4. In the lower right part, an automatically updated copy of the test person’s screen is displayed so that the experimenter can track what the test persons see from their screen.
The second monitor is for the test person. The window on the monitor looks like a one-color screen, where the images with the questions appear alternately (Figure 2), or only a question if there is no image.
Figure 2. View of the exposure window on the monitor for the test person
To synchronize the test person's mental response with the recording of the electroencephalogram, a device has been developed that includes a light sensor mounted on the experimenter's monitor screen opposite the Signal Window (SW). A device with a photosensor is connected to the input of one of the channels of the electroencephalograph.
SW represents light or black squares that replace each other during testing (Figure 3).
Figure 3. Signal window state: 1 - active, 2 - inactive
Inactive SW is black and the photosensor does not respond. As soon as the image with the question on the test person’s monitor screen disappears, SW becomes white on the experimenter's monitor screen, which leads to the appearance of a signal on the photosensor, which is displayed on the electroencephalograph record as a mark.
The program allows you to conduct an experiment in automatic and semi-automatic modes.
In automatic mode, after starting the experiment, the program according to the algorithm specified in the Test Program window performs certain actions for each question, and after the algorithm is completed, it proceeds to the next question and repeats the algorithm for it.
In semi-automatic mode, after each question, the program waits for the experimenter to press the button to move to the next question.
At the beginning of each test, a question and an image appear on the test person’s monitor. The duration of the demonstration can be set in the program (from 1 second, in standard settings the duration of the demonstration is set to 5 seconds). At this time, the SW on the experimenter's monitor screen is black.
Immediately after the image on the test person's screen disappears, they must give a mental answer according to the instructions. The time that is given for a mental answer can be configured in the program (from 1 to 10 seconds, in standard settings the response time is set to 5 seconds).
As soon as the image on the test person’s screen disappears, the SW color on the experimenter's screen instantly changes to white. As a result, the photosensor is triggered and a mark is applied to the channel of the electroencephalograph to synchronize the EEG recording with the mental response of the test person.
The program allows you to randomly generate a set of images and questions that consistently appear on the screen, each time anew. The number and order of questions can be changed using the settings in the Control Panel. It is also possible to change a number of test window parameters: window color, text size and color.
During or after the experiment, the experimenter can use the drop-down list for each question to indicate the result of each test: the test person gave the correct mental answer to the question or could not do it.
After the experiment is completed, its results can be saved in fail CSV file for further processing, for example, in Excel.
The file contains lines where for each question it is noted:
- Question time accurate to ms,
- Consecutive question number,
- Question text,
- Image title,
- The test result - "the correct answer received or not."
The program contains a database of image questions, including 80 questions without images, and 17 portraits of prominent people. Images have enough resolution (300 dpi) and are presented in color. Each drawing is depicted in a colorful and clear manner.
The program is intended to be used on a computer running Windows 7,8,10, equipped with two displays - one for the experimenter, the second one for the test person.
The computer program we created was created using Microsoft Visual Studio 2017, Net Framework 4.5, is easy to use, and has a Russian interface [23] (Figures 1 and 2).
During the tests, the EEG is registered and recorded for further comparative wavelet analysis of the EEG for two mental states of the test person’s brain: extracting information from memory or lack of knowledge.
The test procedure for the test person's mental responses features several steps.
Before testing, electrodes are fixed on the surface of the test person's head to record the EEG in different leads.
The experimenter gives instructions to the test person as following. An image with a question or only a question will appear on the monitor screen in front of the test person. The time for presenting an image with a question for observation to the test persons can be set by the experimenter, and average 5 seconds.
After the image on the screen has disappeared, the test person recalls the answer and, if he/she knows the answer, mentally answers the question to himself/herself. The set response time is set by the experimenter, and is 5 seconds on average.
During testing, the following situations of reproducing information from memory are possible: 1. the test person gave a mentally correct answer to the question posed, and extraction from memory occurred; 2. the test person could not give a mentally correct answer to the question posed because he/she had forgotten the information received earlier; 3. the test person could not give a mentally correct answer to the question posed, because he/she had never obtained such information nor remembered it. During the mental response, an EEG recording is performed. After the test is completed, the test person finds out whether he/she was able to remember and give an answer. This is recorded in the protocol. Then, all files with EEG recording during the mental response will be analyzed using the methods of wavelet transform and machine learning [1-4].
For the first time, a fundamentally new methodology for studying the mechanisms of extracting information from the human brain’s memory was proposed, original PC software was developed, and an experimental setup was created for testing the mental extraction of information from the test person’s brain memory with simultaneous recording of an electroencephalogram (EEG) and subsequent analysis of this EEG using wavelet transform method.
The experimental model and information software make it possible to identify and compare various mental states of the brain's mental activity by electroencephalographic indicators. One state is registered when the test person remembered and gave the correct answer to the question, the other one is registered when the test person did not know or could not remember the answer. These studies are to provide objective indicators of the mental activity of the brain, reflecting the process of recollection from memory.
The disclosure of the mechanisms of the brain's mental activity, including memory, is a fundamental general biological, actual problem of science, the solution of which opens up great opportunities for the development and practical use of fundamentally new information technologies.
This work was supported by grant No. 19-07-00058/19 of the Russian Foundation for Basic Research (RFBR) on the topic: “Research of the possibility of creating an information system for identifying memory reproduction (memorization) processes in the mental activity of the brain based on wavelet analysis of the electroencephalogram”.
- Grubov VV, Sitnikova EYu, Koronovsky AA, Pavlov AN, Khramov AE (2012) Automatic extraction and analysis of oscillatory patterns on non-stationary EEG signals using the wavelet transform and the empirical mode method. Proceedings of the RAS. Physical Series 76: 1520-1523.
- Koronovsky AA, Makarov VA, Pavlov AN, Sitnikova EYu, Khramov AE (2013) Wavelets in neurodynamics and neurophysiology. M Fizmatlit.
- Pavlov AN, Khramov AE, Koronovsky AA, Sitnikova EYu, Makarov VA, et al. (2012) Wavelet analysis in neurodynamics. UFN 182: 905-939.
- Hramov AE, Koronovskii AA, Makarov VA, Pavlov AN, Sitnikova EYu (2015) Wavelets in Neuroscience. Springer.
- Yumatov EA, Khramov AE, Grubov VV, Glazachev OS, Dudnik EN, et al. (2018) The study of the possibility of recognizing the mental activity of the brain based on the wavelet analysis of the electroencephalogram. Biomedical Radioelectronics Mag 4: 3-12.
- Yumatov EA, Hramov AE, Grubov VV, Glazachev OS, Dudnik EN, et al. (2019) Possibility for recognition of psychic brain activity with continuous wavelet analysis of EEG. Journal of Behavioral and Brain Science 9.
- Anokhin PK (1969) The mental form of reality reflection. Collection of works: Leninist theory of reflection and modernity. Edited by T. Pavlova. 1969 Sofia Section 1, Chapter 3, pp: 109.
- Bekhtereva NP, Budzen PV, Gogolitsyn YuL (1977) Brain codes of mental activity. Science.
- Ivanitsky AM (1999) The main mystery of nature: how subjective experiences arise on the basis of brain processes. Psychological Journal 20: 93-104.
- Nagel T (2001) The conceivability of the impossible and the problem of spirit and body. Questions of Philosophy.
- Pavlov IP (1951) Twenty years of objective study of the higher nervous activity (behavior) of animals.
- Popper K (2008) Knowledge and the mind-body problem: In defence of interaction.
- Sudakov KV (2010) System mechanisms of mental activity. Neurology and Psychiatry S.S. Korsakov.
- Crick FC (1995) Why neuroscience may be able to explain consiousness. Scientific Amer.
- Sperry RW (1952) Neurology and the mind-brain problem. Am Sci 40: 291-312.
- Andreev OA, Velichkovsky BB, Popov LI (1987) Memory mechanisms. Science.
- Ashmarin IP (1987) Memory mechanisms - physiology manual. Science.
- Zinchenko TP (2002) Memory in experimental and cognitive psychology. St. Petersburg Piter.
- Rose S (1995) The making of memory: From molecules to mind. M. "Mir”.
- Yumatov EA (2013) The mental activity of the brain is the "key" to cognition. Bulletin of the International Academy of Sciences 1: 35-45.
- Yumatov EA (2019) Remote-field manifestations of mental activity of the brain. Biomedical Radio Electronics Mag 1: 5-13.
- Yumatov EA (2014) To knowledge of the origin of the brain mental activity. World Journal of Neuroscience 4: 170-213.
- MacDonald M (2012) Pro WPF 4.5 in C #: Windows presentation foundation in NET 4.5 Apress, 1095 p. 4th Edition.
