Base64 is an ASCII representation of binary data often used to pass raw binary data in environments that don’t support it, such as SMTP. I covered Base64 encoding earlier and wrote my own C++ implementation of a Base64 encoder in a post, but this article will explain how Base64 decoding works.
Following the Base64 Decoding Process
Let’s decode SGVsbG8gV29ybGQ=.
We start by splitting it up by four-letter groups.
SGVs
bG8g
V29y
bGQ=
The Base64 standard alphabet can be used to convert every letter back into a number. For example, using the first quartet (S, G, V, and s) and dereferencing it against that alphabet gives you 18, 6, 21, and 44.
Then, you take these numbers and convert them to binary which gives you 010010, 000110, 010101, and 101100. Concentrating these binary numbers together (in other words, just combining them) gives us 010010000110010101101100.
Do the same for all groups and add the binary strings together. This will leave us with 0100100001100101011011000110110001101111001000000101011101101111011100100110110001100100.
We are now left with a long string of binary data. Converting that to ASCII gives us the text Hello World.
If the final quartet has padding at the end, the number of padding characters tells you how many bytes in that last quartet contain actual data and not stuff added to pad the encoded string.
One byte of padding (=): output only 2 bytes from the final quartet – the rest is padding
Two bytes (==): output just one byte
None: output all three bytes
Writing a C++ Base64 Decoder
Let’s begin by including iostream for writing to the console.
C++
#include<iostream>
Then for reading and writing files, we can use fstream.
C++
#include<fstream>
After that, we can use cstdint to give us better control over the numbers we store.
C++
#include<cstdint>
For keeping the Base64 input, we can use a string.
C++
#include<string>
Now, let’s define our main function.
C++
intmain(){return0;}
Within our main function, we can hardcode a decoding table for the Base64 standard alphabet.
C++
// constexpr will tell the compiler to include the variable directly in the executable instad of storing it at runtime, which will make the program faster// static will tell the compiler to keep space allocated for the variable throughout the entire program's runtime, which will also make things fasterstaticconstexprunsignedchar padChar ='=';// C++ can easily convert a single letter into a number, so for every numerical representation of a number (according to the ASCII specification), the decode table will have its index in the Base64 alphabet// Here, 0xFF is used to denote a charcater not part of the Base64 alphabetstaticconstexpruint8_t decodeTable[256]={ // 0x00–0x0F0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x10–0x1F0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x20–0x2F ' ' … '/'0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,62,0xFF,0xFF,0xFF,63, // '+'=62 '/'=63 // 0x30–0x3F '0' … '?'52,53,54,55,56,57,58,59,60,61,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x40–0x4F '@' … 'O'0xFF,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14, // 0x50–0x5F 'P' … '_'15,16,17,18,19,20,21,22,23,24,25,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x60–0x6F '`' … 'o'0xFF,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40, // 0x70–0x7F 'p' … DEL41,42,43,44,45,46,47,48,49,50,51,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x80–0x8F0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0x90–0x9F0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xA0–0xAF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xB0–0xBF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xC0–0xCF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xD0–0xDF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xE0–0xEF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF, // 0xF0–0xFF0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF,0xFF};
After that, we can open the input and output files.
C++
std::ifstreamin(inputPath,std::ios::in);if(!in.is_open()){std::cerr <<"Failed to open input file\n";return1;}std::ofstreamout(outputPath,std::ios::binary);if(!out.is_open()){std::cerr <<"Failed to create output file\n";return1;}
Next, we can make variables to keep track of the current quartet we are processing and the current index.
C++
char quartet[4];size_t qIndex =0;
Next, we can enter a loop that will process our quartets.
C++
while(true){}
Within our while loop, we need to keep track of the current character.
C++
char c;if(!in.get(c))break; // in.get(c) will place the next character (reading the file from left to right) into the variable c. If it returns false, we have reached the end of the file and need to break out of our while loop
We then should see if the current character is whitespace (a new line, a space, or a tab) and skip the character’s processing if it is.
C++
// The continue keyword will skip all further instructions in the loop and immediately move to the next iteration (in this case, we are immediately moving to the next character in the file)if(c =='\n'|| c =='\r'|| c ==''|| c =='\t')continue;
We also need to update the quartet variable with the current character.
C++
// Using qIndex++ as an expression like this increments qIndex by 1 and returns the updated valuequartet[qIndex++]= c;
If qIndex is four, we have reached the end of the current quartet and need to process it.
C++
if(qIndex ==4){}
Within this if statement, we can declare a variable to collect four Base64 six-bit chunks before processing them (since Base64 works on groups of six).
C++
uint8_t v[4];
We can also keep track of the padding we have encountered so far.
C++
int padCount =0;
Then we can loop over v and keep track of it against the current quartet.
C++
for(int i =0; i <4; i++){}
Within this loop, we can check if the current character is padding.
C++
if(quartet[i]== padChar){}else{}
Within the first branch of this if statement (the code to be executed if the current character is padding), we can set v at the current index to zero because we will not need to use it later and setting it to zero would be better than undefined garbage data.
C++
v[i]=0;
We can then increment padCount so the next iteration of the for loop can use it.
C++
padCount++;
In the second branch, we can add the Base64 alphabet index to v according to the decoding table.
C++
uint8_t val = decodeTable[(unsignedchar)quartet[i]];if(val ==0xFF){std::cerr <<"Invalid Base64 char\n";return1;} // Earlier, we used 0xFF for characters not part of the Base64 alphabetv[i]= val;
Outside of this if statement and after the for loop, we can take the entire quartet and combine it.
C++
uint32_t triple =(v[0]<<18)|(v[1]<<12)|(v[2]<<6)| v[3]; // We basically concentrate the groups together with the classic binary shift and OR gate method
The padding chooses how many groups are actual binary data and not inserted for correction, so we can extract data from the full triple based on that.
C++
// The reverse binary shift and the AND logic gate is the exact opposite of what we did before. We essentailly "peel apart" the groups based on the number of padding characters.if(padCount <3){char b1 =(triple >>16)&0xFF; // Shifts the top 8 bits (bits 23-16) down into the lowest byte position. Then, we mask with 0xFF so only those 8 bits remain. This gives you the first decoded byte.out.put(b1);std::cout.put(b1);}if(padCount <2){char b2 =(triple >>8)&0xFF; // Shifts bits 15..8 down. Then masks with 0xFF. This is the second decoded byte.out.put(b2);std::cout.put(b2);}if(padCount <1){char b3 = triple &0xFF; // Takes bits 7..0 directly. No mask is needed here because we have reached the end of the triple. That gives the third decoded byte.out.put(b3);std::cout.put(b3);}
Then we can reset qIndex.
C++
qIndex =0;
After the while loop (so we are back in top-level scope now), we can close the input and output files and exit with code zero (for “success”).
This is a follow-up to an earlier post where I explained how Base64 encoding and decoding works. If you have not read that yet, I would recommend you do as it provides helpful background information.
C++ Implementation of Base64 Encoding
Now that we know how Base64 encoding works, we can write our own encoder using C++.
I would like to be able to use files while I am encoding it, so this encoder will store the entire file in a vector, and the decode it from there. This is not really RAM-efficient, however, but I won’t be encoding large files.
First, let’s import our standard IO.
C++
#include<iostream>
Since we will be reading from the file to be encoded and writing to the file that needs to store all of the Base64 as text, we need to import an inbuilt library used for file operations.
C++
#include<fstream>
Since we will be working with bits, we should also be using bit sets in our program. The type needs to be imported from an inbuilt library, however, so let’s import it.
C++
#include<bitset>
Then, since we will be storing the data in a vector, we will need to import that type.
C++
#include<vector>
I want to work with strings, not C-style arrays of characters, so let’s import that.
C++
#include<string>
Because we will be using typedefs (type aliases) like uint8_t that will give us more granular control over how much space each number we store takes up in RAM, we will also need to include cstdint.
C++
#include<cstdint>
Now, we can start with the main function.
C++
intmain(){return0;}
Within this main function, we should first start by asking the user which file they should read. We could use std::cin, but that considers all whitespace characters terminating, meaning that it will not read anything after a whitespace character. So, if there are spaces in our filename, we won’t be able to read it properly.
Instead, we should use std::getline; something like that will only consider newline characters terminating, which is useful because newlines aren’t allowed in file paths anyways.
Below, we collect the input and output files.
C++
// Standard Base64 alphabetstd::string base64Standard ="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; // Values 0-63constchar padChar ="="; // Technically, the padding character is NOT part of the alphabet and so including it above would be misleading// Variable that holds our input filestd::string filePathInput;std::cout <<"Enter a file path to read from\n>";// Read the line and place it directly in filePathInputstd::getline(std::cin, filePathInput);// Variable that holds our output filestd::string filePathOutput;std::cout <<"Enter a file path to write to\n>";// Read the line and place it directly in filePathOutputstd::getline(std::cin, filePathOutput);
After that, we can try opening the file that we need to read from.
C++
std::cout <<"\nAttempting to open input file...\n";// Opening a read-only file stream to whichever path the user specified, in binary mode (since we are reading ones and zeroes, not text)// To be more specific: On Windows systems, binary mode will prevent the OS from turning \n into \r\n, which will mess up the encoding and result in incorrect outputstd::ifstreaminputFileStream(filePathInput,std::ios::binary);
Then, we will see if our file stream has actually been opened. If it hasn’t, that likely means some error occurred that prevented the file from opening.
C++
if(!inputFileStream.is_open()){std::cerr <<"Failed to open file!\n";return1;}std::cout <<"File open!\n";
In order to maintain speed, we should pre-allocate the file data that will be stored in a vector. To do this, we seek to the end of the file, and then use the tellg function to tell us how far we have seeked from the beginning of the file. Then, we store this a variable.
C++
std::cout <<"Detecting file size...\n";inputFileStream.seekg(0,std::ios::end);// There is a dedicated type for file stream positions in C++conststd::streampos endPos = inputFileStream.tellg();if(endPos <0){std::cerr <<"Failed to get file size\n";return1;}constsize_t fileSizeBytes =static_cast<size_t>(endPos); // We must convert std::streampos to the dedicated type I mentioned above, size_tstd::cout <<"Detected file size: "<< fileSizeBytes <<" bytes\n";inputFileStream.seekg(0,std::ios::beg);
Now that the file stream is open, we can load the entire thing into a vector, and then close it as we have no use for it anymore.
C++
// Each character in the vector holds raw bytes of the filestd::vector<uint8_t>fileData(fileSizeBytes);std::cout <<"Reading file...\n";// read(...) takes char* and std::streamsize.// In C++, a static cast is a cast (type conversion) where there is native support for the two types to convert from and to each other// We use one here to convert the result of fileData.size() into std::streamsize// A reinterpret cast tells the compiler that we know for sure the data coming in can be converted into a certain type (std::streamsize in this case)// This essentially causes the type conversion to happen during runtime// This is very unsafe and is not usually recommendedinputFileStream.read(reinterpret_cast<char*>(fileData.data()),static_cast<std::streamsize>(fileData.size()));inputFileStream.close();std::cout <<"File data read! It is now safe to modify or delete the file!\n";
After that, we can create the output file.
C++
std::cout <<"Creating output file...\n";std::ofstreamencodedDataOutputFile(filePathOutput,std::ios::binary);// Fires if an error occurs during the file creation processif(!encodedDataOutputFile.is_open()){std::cerr<<"Error opening output file\n";return1;}
Now, we can process each and every bit to convert it into a Base64 character.
C++
size_t totalBits = fileData.size()*8; // There are 8 bits in a bytestd::cout <<"Processing "<< totalBits <<" bits...\n";size_t bitsProcessed =0; // Let's keep track of the bits processed herewhile(bitsProcessed < totalBits){}
Within this while loop comes the actual processing of bits.
Let’s start by keeping track of the current position of the byte being processed and the bit being processed, which will be useful for future calculations.
Keeping track of the byte index is simple – you just divide the number of bits processed by 8, since there are 8 bits in a byte.
Keeping track of the bit index required some more thinking for me. We need to use the bit index to determine the number of bits that spill over. Finding the remainder of the number of bits processed should be good enough for this use case.
C++
size_t byteIndex = bitsProcessed /8;size_t bitIndex = bitsProcessed %8; // This tells us how many bits are overflowing into the next byte
Next, we will keep track of the current byte and the next byte, since it is possible for six bits to span across two bytes.
C++
uint8_t currentByte = fileData[byteIndex];uint8_t nextByte =(byteIndex +1< fileData.size())? fileData[byteIndex +1]:0; // Six bits may span across two bytes, so we need to keep track of the next byte
Next, we extract six bits to cross-reference it to our alphabet.
To do this, we put two bytes (16 bits), store it in a variable called combined, and extract six bits from it. We do it this way because, like I said before, six bits can span across two bytes.
Last but not least, we output this character to both the console and the output file the user specified at the beginning and increment the bitsProcessed by 6.
C++
encodedDataOutputFile << resultChar;std::cout << resultChar;bitsProcessed +=6; // Advance the loop
C++ Implementation: Solving the Padding Issues
Now, we just need to add padding to our C++ program.
Keep in mind that we have now moved on from the while loop, and any future snippets of code will take place in the main function.
Let’s start by computing the amount of padding characters we need using the implementation I showed you at the beginning of this tutorial.
C++
size_t remaining = fileData.size() /*The size of the input file, in bits*/%3;size_t padCount =(3- remaining)%3; // yields 0, 1, or 2
Then, we add however many padding characters we need to the console and the output file.
C++
for(size_t i =0; i < padCount;++i){ encodedDataOutputFile << padChar;std::cout << padChar;}
Final Code
And we are done! Your code should look something like this:
C++
#include<iostream>#include<fstream>#include<cstdint>#include<bitset>#include<vector>#include<string>intmain(){ // Standard Base64 alphabetstd::string base64Standard ="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; // Values 0-63constchar* padChar ="="; // Technically, the padding character is NOT part of the alphabet and so including it above would be misleading // Variable that holds our input filestd::string filePathInput;std::cout <<"Enter a file path to read from\n>"; // Read the line and place it directly in filePathInputstd::getline(std::cin, filePathInput); // Variable that holds our output filestd::string filePathOutput;std::cout <<"Enter a file path to write to\n>"; // Read the line and place it directly in filePathOutputstd::getline(std::cin, filePathOutput);std::cout <<"\nAttempting to open input file...\n"; // Opening a read-only file stream to whichever path the user specified, in binary mode (since we are reading ones and zeroes, not text) // To be more specific: On Windows systems, binary mode will prevent the OS from turning \n into \r\n, which will mess up the encoding and result in incorrect outputstd::ifstream inputFileStream(filePathInput,std::ios::binary);if(!inputFileStream.is_open()){std::cerr <<"Failed to open file!\n";return1;}std::cout <<"File open!\n";std::cout <<"Detecting file size...\n"; inputFileStream.seekg(0,std::ios::end); // There is a dedicated type for file stream positions in C++conststd::streampos endPos = inputFileStream.tellg();if(endPos <0){std::cerr <<"Failed to get file size\n";return1;}constsize_t fileSizeBytes =static_cast<size_t>(endPos); // We must convert std::streampos to the dedicated type I mentioned above, size_tstd::cout <<"Detected file size: "<< fileSizeBytes <<" bytes\n"; inputFileStream.seekg(0,std::ios::beg); // Each character in the vector holds raw bytes of the filestd::vector <uint8_t>fileData(fileSizeBytes);std::cout <<"Reading file...\n"; // read(...) takes char* and std::streamsize. // In C++, a static cast is a cast (type conversion) where there is native support for the two types to convert from and to each other // We use one here to convert the result of fileData.size() into std::streamsize // A reinterpret cast tells the compiler that we know for sure the data coming in can be converted into a certain type (std::streamsize in this case) // This essentially causes the type conversion to happen during runtime // This is very unsafe and is not usually recommended inputFileStream.read(reinterpret_cast<char*>(fileData.data()),static_cast<std::streamsize >(fileData.size())); inputFileStream.close();std::cout <<"File data read! It is now safe to modify or delete the file!\n";std::cout <<"Creating output file...\n";std::ofstream encodedDataOutputFile(filePathOutput,std::ios::binary); // Fires if an error occurs during the file creation processif(!encodedDataOutputFile.is_open()){std::cerr <<"Error opening output file\n";return1;}size_t totalBits = fileData.size()*8; // There are 8 bits in a bytestd::cout <<"Processing "<< totalBits <<" bits...\n";size_t bitsProcessed =0; // Let's keep track of the bits processed herewhile(bitsProcessed < totalBits){size_t byteIndex = bitsProcessed /8;size_t bitIndex = bitsProcessed %8; // This tells us how many bits are overflowing into the next byteuint8_t currentByte =fileData[byteIndex];uint8_t nextByte =(byteIndex +1<fileData.size())?fileData[byteIndex +1]:0; // Six bits may span across two bytes, so we need to keep track of the next byteuint16_t combined =(static_cast<uint16_t>(currentByte)<<8)| nextByte;uint8_t sixBits =(combined >>(10- bitIndex))&0x3F; // Extracting six bitschar resultChar =base64Standard[static_cast<int>(sixBits)]; encodedDataOutputFile << resultChar;std::cout << resultChar; bitsProcessed +=6; // Advance the loop}size_t remaining = fileData.size() /* <-- The size of the input file, in bits */%3;size_t padCount =(3- remaining)%3; // yields 0, 1, or 2 (for number of padding characters we need to add)for(size_t i =0; i < padCount;++i){ encodedDataOutputFile << padChar;std::cout << padChar;}return0;}
Performance Testing
I wanted to compare this implementation to [Convert]::ToBase64String on PowerShell.
Disappointing but not at all surprising. Not only is our implementation made for readability, not efficiency, our implementation writes to a file (which is a lot slower than you think).
To begin, I added these lines right after we declare the top menu bar container.
index.html
<divid="top_menu_options"><!-- PLACEHOLDER TEXT: --> Lorem ipsum dolor sit amet consectetur, adipisicing elit. Qui, nobis accusantium. Blanditiis similique, architecto porro quod consequuntur consectetur, est culpa ad deserunt eveniet, eius laborum! Voluptatum, sapiente? Labore, ut et.</div>
To give it some style, we can start a new CSS ruleset. Let’s make it float above the other elements and have a minimum width and height of 200 pixels, a preferred width of 20% viewport width, a maximum width of 300 pixels, a maximum height of 500 pixels, and a preferred height of 30% viewport height.
Let’s give it a background blur and a background color. It should change with the theme, so let’s add a variable to our .dark ruleset.
style.css
.dark {/* We must write it this way so that we can give it a background blue with backdrop-filter: blur(...) and rgba(...) */--context-bg-color: 49, 49, 49; /*#313131*/}
Then we can apply this color (mixed with some transparency) and a background blur to our context menu.
Within this script, I put some starter code in that would allow us to log to the console while telling us where the message was coming from and loop over every menu item to give them click and mouseenter listeners.
In the future, it will come in handy to keep track of the context menu state. That is why we are going to add this line right after we declare the log function.
top_menu_bar.js
constcontext_menu_state = { clicked:"" }; // I can already tell that we will need to keep track of the clicked item, so I'm adding it here now
We should also set the actual context menu itself to a variable.
Because we will be keeping context_menu_state['clicked'] as a string for easy debugging, we should probably link those strings to actual HTML elements we can manipulate in the DOM.
top_menu_bar.js
constmenu_bar_items = {}; // We will populate this later. We should be focused on scalability, so using preset menu items would become more tedious in the long run
Then, we can populate this object.
top_menu_bar.js
Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item=> { // We have to convert it to an array to run forEach on it.menu_bar_items[top_menu_item.innerText] = top_menu_item;});
Showing the Context Menu on Click
Let’s get back to our click listener in the loop over every top menu item and show the context menu under it.
top_menu_bar.js
context_menu_container.style.display = "block";context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";context_menu_state.clicked = top_menu_item.innerText; // We declared this object earlier, now we need to update it
Let’s also give an indication of the selected item in the menu bar by adding a CSS class to it so we can style it later. This also involves first removing it from any other menu items.
However, inspecting the top menu items gives us a problem. The mouseenter event does not trigger for HTML margins, so the spaces between the menu items won’t trigger an event. This can be fixed by nesting the text in a p element, like this.
In a menu bar, if you click on a menu item and move your mouse over to another item, the options move from the original one you clicked and appear under that item. Let’s recreate this by adding code to the inside of our mouseenter event. I am going to avoid mouseover since that fires multiple times if your cursor is in the element, not just when your cursor enters the bounds of the element.
Within this event, we first need to make sure that something was clicked before we show the context menu.
top_menu_bar.js
if (context_menu_state.clicked !== "") {}
Then within this if statement, we can make sure that the context menu is visible and then make it appear under the currently hovered menu item.
We should also give a visual indication of the currently hovered item, so let’s add a class to it called force_white_border_underline. This class should create a distinction between the currently hovered item and the currently selected item. We should only add it if the selected element actually does differ from the hovered element, though.
Hiding the Menu Options Whenever the User Clicks Elsewhere
We now have a fully functional menu bar system, unless you want to leave the menu bar and exit completely. Sure, we could use focusout, but the menu bar items aren’t even allowed to have focus, and even if we set their tabindex to zero, clicking on a context menu will instantly hide it, since a menu bar item lost focus. On most Windows systems, this is the case, however what if in the future I decide that the context menu options should remain visible until after the action has completed. The goal here is to get the most flexibility.
To accomplish this, we will use a click event on all of the document (since the menu options are outside the scope of the top menu items) and then hide and remove the classes we may have added to it if the click event was not fired because the user clicked on another menu bar item (or its children) or the context menu and its children.
This code to add the event listener will not be added in any scopes, at the every end of the file.
In order to gain more individual control over each menu option, there will be an unordered list within this context menu container, and each list item will be an option of the menu. It will also be a flex container (which will help with responsive layouts and horizontal alignment later). Each list item will have two more flex containers inside of it, one aligned to the left and one to the right (we will use justify-content: space-between; on the list item to accomplish this). For now, the left div will just contain the action while the right will contain the keyboard shortcut or an arrow, if the action is meant to trigger a submenu.
Below is an example of the structure. We can add more list items later.
index.html
<divid="top_menu_options"class="comfy_spacing"><!-- .comfy-spacing will be useful for customization later --><ulid="top_menu_options_ul"class="undecorated_list"><liclass="top_menu_options_li flex_container"><divclass="flex_container"><pclass="top_menu_options_choice_inner_p_main_text">New</p></div><divclass="flex_container"><pclass="top_menu_options_choice_inner_subtext">Ctrl+N</p></div></li><liclass="top_menu_options_li flex_container"><divclass="flex_container"><pclass="top_menu_options_choice_inner_p_main_text">Open</p></div><divclass="flex_container"><pclass="top_menu_options_choice_inner_subtext">Ctrl+O</p></div></li></ul></div>
Then, we can implement some basic styling in CSS.
style.css
.undecorated_list {list-style-type: none;padding: 0;margin: 0;}#top_menu_options_ul {height: 96%;width: 94%;padding-left: 3%;padding-right: 3%;padding-top: 2%;padding-bottom: 2%;overflow-y: scroll;scrollbar-width: none;}#top_menu_options_ulp {margin: 0;cursor: default;}#top_menu_options_ulli {justify-content: space-between;}#top_menu_options_ullip.top_menu_options_choice_inner_subtext:not(p.disregard_subtext_formatting) { /*We don't want to make the right arrow dimmed and italic*/color: var(--subtle-color);font-style: italic;}#top_menu_options_ulli:not(#top_menu_options_ulli:first-child) {margin-top: clamp(var(--spacing-between-menu-items-min),var(--spacing-between-menu-items-preferred),var(--spacing-between-menu-items-max));}#top_menu_options_ulli:not(.not_a_context_menu_option):hover { /*This may be useful for labels and submenu headings within the context menu*/font-weight: bold;}#top_menu_options_ulli:not(.not_a_context_menu_option):hoverp.top_menu_options_choice_inner_p_main_text {color: color-mix(insrgb, rgb(var(--accent-1)) 90%, var(--text-color) 10%);}.context_menu_header { /*Again, useful for submenu headings*/border-bottom: 2pxsolidvar(--subtle-color);padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));}
You may notice that we have three variables referenced here, namely --spacing-between-menu-items-preferred, --spacing-between-menu-items-min, and --spacing-between-menu-items-max. Before this CSS code, we first need to define these variables.
style.css
.dark {--subtle-color: rgb(133, 133, 133); /*This should change with theme, and that is why it was referenced above*/}.compact_spacing {--spacing-between-menu-items-min: 0px;--spacing-between-menu-items-preferred: 0px;--spacing-between-menu-items-max: 0px;}.comfy_spacing {--spacing-between-menu-items-min: 0px;--spacing-between-menu-items-preferred: 10px;--spacing-between-menu-items-max: 100vh;}
In the future, we can change the class to increase or decrease the spacing between menu items.
Making the Buttons Do Stuff
To change the buttons for every menu item you click and make them do things when you click it, I made a separate file for this called top_menu_options.js.
In this file, I defined the log function and a placeholder function for things that will be implemented later. Then, added a reference to the inner unordered list of the top menu options in the DOM.
Then I came up with a structure for declaring the context menu items as an object.
Each key in this object corresponds to the text of the menu item that should be clicked for these specific items to show up. Every value is array of objects, with the objects this time representing each item in the context menu. There can be many different kinds of things you would want to put in here, so make sure to mark each of them with a type. I am using numbers to represent the types, and I am putting all the possible options in a JSDoc comment.
top_menu_options.js
/** * An object of menu options. For every option, there is another array of options (represented by objects). Below are the parameters each one can take. * - `type` (integer): Represents the type of the item * - `0`: Represents a menu label with text. It is used to, for example, display the submenu breadcrumbs. It is only recommended to be used as the first menu item. * - `1`: Same as `0`, but with no text. This makes it just a separator. * - `2`: A regular menu option that can be clicked to preform an action. * - `3`: A submenu that can be clicked to open up more options. * - `4`: Back header (used for properly displaying context menu breadcrumbs) * - `5`: A checkbox item that can be toggled to on or off. The function given to `handler` is called every time it is clicked with a single argument passed to it - whether if the item has been changed to checked or not * - `content` (string): The **unsanitized** HTML to display (not used on type `1`) * - Note: on type `4` items, all the previous breadcrumbs will be dimmed and this will be displayed on the right side of the menu options * - `subtext` (string): The **unsanitized** HTML to display on the right side in a darker color, typically a keyboard shortcut (only used on `2`) * - `openTo` (array[object]): Another array of menu item objects that will be opened up when the current option is clicked (only used on `3`) * - `handler` (function): A function to be run when the menu option is clicked (only used on `2` and `5`) * - `checked` (boolean): Whether the menu item is checked by default or not (only used on `5`) * - Note: This value changes. It is used by FeatherText to keep track of the current item's state * - `disabled` (boolean): Whether the current option is disabled and cannot be used in the current scenario (only used on `2` and `5`) * - Note: This is automatically enabled on type `3` items if all the type `2` items in `openTo` have this enabled. This cannot be set manually on type `3` items. * - Note: If this is enabled on a type `5` item, it will remain locked in the current state whether it is checked or not */
Then right after, I begin defining the menu items.
Let’s think ahead for a moment. If we want to open up a submenu, we will need to call the function to redraw the menu with a custom array and keep track of the previous items that the menu was opened to. Let’s define those two things.
We don’t want top and bottom borders being applied if it is either the first or last elements, so let’s apply a different class for each scenario.
Scenario 1: The context menu header is the first item
top_menu_options.js
if (idx===0) {new_list_item.classList.add("context_menu_header")}
Scenario 2: The context menu header is the last item
top_menu_options.js
elseif (idx===items.length-1) { // idx is 0-indexed, while Array.prototype.length is not. This is why we subtract 1new_list_item.classList.add("context_menu_ending_header")}
Scenario 3: Neither of the conditions above are true, meaning that the header is in the middle of the context menu.
Type one items are simply context menu separators, so this one is fairly straightforward. All code in this section will be contained inside our second if clause.
Let’s start by making a new list item.
top_menu_options.js
constnew_list_item = document.createElement("li")
It isn’t an actual clickable menu item, so let’s add some identifiers to reflect these properties in the CSS we will add later.
Type two items are probably the most common type I will use because this represents a regular item that you can click on to perform an action. The code here will be part of the third clause.
Let’s start by making a new list item, adding the relevant classes to it, and then attaching it to the handler.
Type three items are the buttons that lead to a submenu. Rather than draw a new box, I think it would be better to do what Firefox does and show the new menu options without spawning a new box. All code will be in the fourth if clause.
We can structure this the exact same as type two items, but they do a different thing when they are clicked. We also want to stop the regular subtext formatting from being applied to this element.
Then for the right-side text, we will do something a little different. We will use a right quote (›) character instead of subtext defined in the menu_options object. We will also prevent the regular subtext formatting from being applied by adding another class to this text, disregard_subtext_formatting.
Then, we can handle a click event by stopping all other events (to make sure that the menu item does not lose focus and disappear), and setting the current array as the one previously opened because in the next line, we redraw the context menu items with the elements that are supposed to be within the submenu.
Type four items are the headers that allow you to open a submenu’s parent. All code in this section will be in the fifth clause.
Because we need to make sure this option is as visible as possible to the user in case they clicked on the submenu by mistake, it must be the first item in the list of context menu items. We can run a quick check to see if this is true, after which we will log an error message and skip drawing this item if it is not.
top_menu_options.js
if (idx !== 0) {log("Type 4 elements must be the first item of the array!");continue;}
Then, we can make a list item and have it show the previously opened menu on click.
For the right-side text, we are going to do something interesting. We are going to make the element, as usual, and then dim its content, except for the final slash. We will do this by using a regular expression to wrap everything that needs to be dimmed in a span tag.
top_menu_options.js
constright_side_text = document.createElement("p")right_side_text.innerHTML = element['content'].replace(/^(.*\/)/, '<span style="color: var(--subtle-color);">$1</span>');right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting") // We want to remove as much default CSS styling as possible, which is why we do not apply the normal formatting for subtextright_side_div.appendChild(right_side_text)
Constructing Type Five Items
Type five items are the checkbox items that turn something on or off. All code in this section will be in the sixth and final if clause.
We can define the new list item along with the left and right side divs and texts like we would normally do.
Before the text on the left side, we should put a box to indicate whether or not the current option is selected. This will be a div element that will be filled if the option is selected, and only show the border if the option is not selected.
Then, using CSS, we can make the context menu indicator however we want. In this case, it will be a small div that can be filled to indicate that the menu option is selected, or empty to indicate that the menu option is not selected.
style.css
.context_menu_option_selection_indicator {height: 9px; aspect-ratio: 1; /* Makes the height and width equal */align-self: center; /* Centers itself vertically */border: 2pxsolidvar(--text-color);margin-right: 8px; /* Gives space between the indicator and the text */}.context_menu_option_selection_indicator_selected {background-color: var(--text-color);}.context_menu_option_disabled.context_menu_option_selection_indicator {border-color: var(--subtle-color);}.context_menu_option_disabled.context_menu_option_selection_indicator_selected {background-color: var(--subtle-color) !important;}
Redrawing The Menu Items Based on a Menu Option
In order to do this, we can draw the menu options for that specified menu item.
top_menu_options.js
exportfunctionredraw_menu_options(menu_item) {if (menu_options[menu_item] === undefined) {log("Not a valid menu option!");return;}redraw_menu_from_array(menu_options[menu_item])}
Structuring The Program
Notice how we added export to the beginning of the function declaration. This is because of the way I am going to structure this program. We will import the function from this file and use it in top_menu_bar.js. This code will be at the beginning of top_menu_bar.js. I will be moving both of those files into a separate subfolder called top_menu.
In top_menu_options.js, reset_previous_open_to is not exported from the file. Let’s change that by adding the export keyword to the function declaration at the beginning of the file.
Going back to top_menu_bar.js, we can now put the following line as the final line before we close the click and mouseenter listeners in our menu item loop.
top_menu_bar.js
redraw_menu_options(top_menu_item.innerText)
Then at the click for the entire document, we can put this line right before we close the if statement.
top_menu_bar.js
reset_previous_open_to()
Then, I renamed top_menu_bar.js to top_menu.js, so that our directory tree looks like this.
functionlog(txt) {console.log('[top_menu.js][top_menu_options.js] ', txt);}log("Script loaded!");constno_op = ()=>{}consttop_menu_options_ul = document.getElementById("top_menu_options_ul")/** * An object of menu options. For every option, there is another array of options (represented by objects). Below are the parameters each one can take. * - `type` (integer): Represents the type of the item * - `0`: Represents a menu label with text. It is used to, for example, display the submenu breadcrumbs. It is only recommended to be used as the first menu item. * - `1`: Same as `0`, but with no text. This makes it just a separator. * - `2`: A regular menu option that can be clicked to preform an action. * - `3`: A submenu that can be clicked to open up more options. * - `4`: Back header (used for properly displaying context menu breadcrumbs) * - `5`: A checkbox item that can be toggled to on or off. The function given to `handler` is called every time it is clicked with a single argument passed to it - whether if the item has been changed to checked or not * - `content` (string): The **unsanitized** HTML to display (not used on type `1`) * - Note: on type `4` items, all the previous breadcrumbs will be dimmed and this will be displayed on the right side of the menu options * - `subtext` (string): The **unsanitized** HTML to display on the right side in a darker color, typically a keyboard shortcut (only used on `2`) * - `openTo` (array[object]): Another array of menu item objects that will be opened up when the current option is clicked (only used on `3`) * - `handler` (function): A function to be run when the menu option is clicked (only used on `2` and `5`) * - `checked` (boolean): Whether the menu item is checked by default or not (only used on `5`) * - Note: This value changes. It is used by FeatherText to keep track of the current item's state * - `disabled` (boolean): Whether the current option is disabled and cannot be used in the current scenario (only used on `2` and `5`) * - Note: This is automatically enabled on type `3` items if all the type `2` items in `openTo` have this enabled. This cannot be set manually on type `3` items. * - Note: If this is enabled on a type `5` item, it will remain locked in the current state whether it is checked or not. It can be changed programatically, but not by the user */constmenu_options = {"File":[],"Edit":[],"Go To":[],"Cursor":[],"View":[],"Options":[],"Help":[]}letprevious_open_to = {}exportfunctionreset_previous_open_to() {previous_open_to={}}functionredraw_menu_from_array(arr) {// Clear existing itemstop_menu_options_ul.innerHTML = ''// Add new itemsfor (const [idx, element] ofarr.entries()) {if (element['type'] === 0) {constnew_list_item = document.createElement("li")new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option")if (idx===0) {new_list_item.classList.add("context_menu_header") } elseif (idx===items.length-1) {new_list_item.classList.add("context_menu_ending_header") } else {new_list_item.classList.add("context_menu_midway_header") }top_menu_options_ul.appendChild(new_list_item)constnew_list_header_text = document.createElement("p")new_list_header_text.classList.add("top_menu_options_choice_inner_p_main_text")new_list_header_text.innerHTML = element['content']new_list_item.appendChild(new_list_header_text) } elseif (element['type'] === 1) {constnew_list_item = document.createElement("li")new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option", "context_menu_separator")top_menu_options_ul.appendChild(new_list_item) } elseif (element['type'] === 2) {constnew_list_item = document.createElement("li")new_list_item.classList.add("top_menu_options_li", "flex_container")if (!element['disabled']) {new_list_item.addEventListener("click", element['handler'])}top_menu_options_ul.appendChild(new_list_item)constleft_side_div = document.createElement("div")left_side_div.classList.add("flex_container")new_list_item.appendChild(left_side_div)constleft_side_text = document.createElement("p")left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")left_side_text.innerHTML = element['content']left_side_div.appendChild(left_side_text)constright_side_div = document.createElement("div")right_side_div.classList.add("flex_container")new_list_item.appendChild(right_side_div)constright_side_text = document.createElement("p")right_side_text.innerHTML = element['subtext']right_side_text.classList.add("top_menu_options_choice_inner_subtext")right_side_div.appendChild(right_side_text)if (element['disabled']) {new_list_item.classList.add("context_menu_option_disabled") } } elseif (element['type'] === 3) {constnew_list_item = document.createElement("li")new_list_item.classList.add("top_menu_options_li", "flex_container")top_menu_options_ul.appendChild(new_list_item)constleft_side_div = document.createElement("div")left_side_div.classList.add("flex_container")new_list_item.appendChild(left_side_div)constleft_side_text = document.createElement("p")left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")left_side_text.innerHTML = element['content']left_side_div.appendChild(left_side_text)constright_side_div = document.createElement("div")right_side_div.classList.add("flex_container")new_list_item.appendChild(right_side_div)constright_side_text = document.createElement("p")right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting")right_side_text.innerHTML = '›'right_side_div.appendChild(right_side_text)new_list_item.addEventListener("click",e=>{e.stopImmediatePropagation()previous_open_to = arrredraw_menu_from_array(element['openTo']) }) } elseif (element['type'] === 4) {if (idx !== 0) {log("Type 4 elements must be the first item of the array!");continue;}constnew_list_item = document.createElement("li")new_list_item.addEventListener("click",e=>{e.stopImmediatePropagation()redraw_menu_from_array(previous_open_to)previous_open_to = arr })new_list_item.classList.add("top_menu_options_li", "flex_container")top_menu_options_ul.appendChild(new_list_item)constleft_side_div = document.createElement("div")left_side_div.classList.add("flex_container")new_list_item.appendChild(left_side_div)constleft_side_text = document.createElement("p")left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")left_side_text.innerHTML = '‹ Back'left_side_div.appendChild(left_side_text)constright_side_div = document.createElement("div")right_side_div.classList.add("flex_container")new_list_item.appendChild(right_side_div)constright_side_text = document.createElement("p")right_side_text.innerHTML = element['content'].replace(/^(.*\/)/, '<span style="color: var(--subtle-color);">$1</span>');right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting")right_side_div.appendChild(right_side_text) } elseif (element['type'] === 5) {constnew_list_item = document.createElement("li")new_list_item.classList.add("top_menu_options_li", "flex_container")top_menu_options_ul.appendChild(new_list_item)constleft_side_div = document.createElement("div")left_side_div.classList.add("flex_container")new_list_item.appendChild(left_side_div)constleft_side_text = document.createElement("p")left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")left_side_text.innerHTML = element['content']left_side_div.appendChild(left_side_text)constleft_side_selection_indicator = document.createElement("div")left_side_selection_indicator.classList.add("context_menu_option_selection_indicator")if (element['checked']) {left_side_selection_indicator.classList.add("context_menu_option_selection_indicator_selected")}left_side_text.insertAdjacentElement("beforebegin",left_side_selection_indicator)constright_side_div = document.createElement("div")right_side_div.classList.add("flex_container")new_list_item.appendChild(right_side_div)constright_side_text = document.createElement("p")right_side_text.innerHTML = element['subtext']right_side_text.classList.add("top_menu_options_choice_inner_subtext")right_side_div.appendChild(right_side_text)if (element['disabled']) {new_list_item.classList.add("context_menu_option_disabled"); }new_list_item.addEventListener("click", ()=>{if (!element['disabled']) {element['checked'] = !element['checked']left_side_selection_indicator.classList.toggle("context_menu_option_selection_indicator_selected")element['handler'](element['checked']) } }) } }}exportfunctionredraw_menu_options(menu_item) {if (menu_options[menu_item] === undefined) {log("Not a valid menu option!");return;}redraw_menu_from_array(menu_options[menu_item])}redraw_menu_options("File")
When you are looking to quickly edit a file, chances are you open something like Notepad or Notepad++. However, on more limiting devices such as a phone or a Chromebook with no built-in text editor (Google Docs is a word processor, so it doesn’t count), your best bet is searching something for an online text editor. The issue with this is that most online text editors are sloppy and don’t get the same features as you do on desktop (and the ones that do feel very overwhelming)
The Idea
Here, I will be refreshing my memory on the three web technologies by creating a text editor that is reminiscent of Vim and Emacs. It should have a tab bar like Emacs, the commands of Vim, and full keyboard support like both. Supporting Vim Script or the macros that gives Emacs its name is out of the scope of this guide, especially for an online text editor. However, they may be supported in a future update.
I will also be using pure HTML, CSS, and JS. No preprocessors or frameworks used here. This is because I am documenting my learning process here for those three technologies, not for external UI frameworks.
One important thing to note is that this text editor will only have two modes, “insert” and “view,” and the default mode will be insert. This is because while I do want to make sure my editor is powerful and familiar to Vim and Emacs users alike, it should also be user-friendly. In insert mode, you can edit the text directly, but if you press ESC and end up in viewing mode, you can use Vim-like commands and control the top menu bar with your keyboard, like you can on Emacs.
Starting Off
I started off with a single file, index.html, where I put in the standard boilerplate and then set the title of the webpage to what this text editor will be named: FeatherText.
Then I linked a CSS file called style.css.
index.html
<!-- Place within <head> tags --><linkrel="stylesheet"href="style.css">
Within that style.css file, I then started by using CSS variables to define a starting theme (did I mention that this text editor will have multiple themes?)
Then, I had to link the theme to the webpage by adding the dark class to the body element. This way, the dark theme will be enabled by default.
index.html
<!-- Replace "<body>" with the text below --><bodyclass="dark">
Afterwards, I removed the scrollbars (since they were just going to cause problems down the line) and applied the dark theme to the webpage by setting the background color and text color to match with the one defined in the theme.
style.css
body {background-color: var(--window-bg-color);color: var(--text-color);height: 100vh;width: 100vw;margin: 0;overflow: hidden;}
Then, I added two fonts to be applied to the entire webpage in different UI elements.
style.css
/* PUT THESE LINES AT THE VERY TOP OF STYLE.CSS *//* Import "Azeret Mono" font (to be used on UI elements like the menu bar) */@importurl('https://fonts.googleapis.com/css2?family=Azeret+Mono:ital,wght@0,100..900;1,100..900&display=swap');/* Import "DM Mono" font (to be used in the text box) */@importurl('https://fonts.googleapis.com/css2?family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&display=swap');
Top Menu Bar
To implement a top menu bar, similar to the one you get in Emacs, I started off with a div full of other divs (this time our menu items), so our HTML file would look something like this:
Now, let’s give each item (except for the first) a minimum of 23 pixels of left margin, but preferably 1.5% of the total width of the screen. Let’s give our first item 10 pixels of left margin.
Let’s give every item a top margin of 10 pixels and clear out any other default margins set by the browser, while making sure that our cursor remains the default (this is just in case we decide to add a link to the top menu bar, which would change the cursor to a pointer).
But wait! You may notice a problem with what I have done. While nothing seems wrong, try putting some sample text below the menu bar and see what happens below.
You probably noticed that the bottom border transition shifted the text downwards, which I would say is pretty distracting and does not look good at all. This is because the border-bottom property affects the size of the element. There are some ways to get the same effect without affecting layout, but the simplest one to explain (and the one I am using here) is to keep the border on all the time but only show it on hover.
The clip-path property essentially allows you to crop an element to a specific viewable portion. Better yet – you can animate it.
What we will be doing is keeping the border always on but set the clip-path property such that the border is hidden. Then, we can animate the clip-path property to show the full shape on hover. Let’s replace our CSS rulesets to use this approach.
style.css
#top_menu_containerdiv.top_menu_item {clip-path: polygon(/* Top left corner */00,/* Top right corner */100%0,/* Bottom right corner */100%calc(100% - 4px),/* Bottom left corner */0calc(100% - 4px) /* Element full width, minus the border width*/ );transition: clip-path 50mslinear; /* Animate it */border-bottom: 4pxsolidvar(--text-color); }
You will notice that the first number of every pair passed into the polygon function is 0, except for the top and bottom right corners, which are 100%. This represents the x-index for our polygon, but because we are not cropping anything from the sides, this can represent the full width.
And then on hover, we show the full shape to reveal the border.
These menu buttons are cool, but they don’t do anything yet. So, I created a folder called scripts and a file in that folder called top_menu_bar_options.js.
Right now, we have the basic look and feel of a traditional menu bar app, but to make it functional, we will need to use some JavaScript. Stay tuned for a part 2!
Some things you might have heard being thrown around in the cryptography space are encoding methods. Such encoding methods are normally used to turn binary data into text, typically so it can be used to transmit data over communication methods that do not allow raw binary data to be transmitted.
How does binary turn into Base64?
First, let’s say that we want to encode an image. We start by reading all the bits in the image, which, for this example, will result in a binary stream of 010010101101001000111111. It is important to note that most images are a lot larger than this and will contain hundreds of thousands of bits.
The encoder will first break the binary stream up into their own chunks, each containing six bits. We can do this to our example and get 010010 101101 001000 111111.
Then, we turn our binary chunks into decimals.
To do this, working right to left, we multiply each bit by 2 to the power of p, where p starts from zero and increments by one for every bit in the chunk.
For example, we can calculate 010010 by performing the expression (0 * 20) + (1 * 21) + (0 * 22) + (0 * 23) + (1 * 24) + (0 * 25). We can strip away any parts that multiply by zero, since those will always be zero. The simplified equation is (1 * 21) + (1 * 24) = 2 + 16 = 18. Therefore, 010010 is equivalent to the decimal 18.
Turning every single six-bit binary chunk into a decimal using the process shown above constitutes the numbers (in left-to-right order) 18, 45, 8, and 63.
After that, we can simply cross reference the numbers with whatever alphabet we should use. For example, the six-bit chunk 010010 turns into the decimal 18, which corresponds to S on the Base64 standard alphabet. Therefore, 010010 101101 001000 111111 turns into StI/.
How does binary turn into Base64?: fixing the padding
If the binary string is a multiple of 3, everything fits perfectly. However, if it is not, then we need to add padding to the encoded string until the string’s bit count becomes a multiple of 3.
Let’s take 010010 101101 001000 111111 01 as an example. It is 26 bits long and all full six-bit chunks encode to StI/. However, this is leaving out the two bits at the end.
To pad this, we will keep adding zeroes to the end of the string until it can become a full chunk. Thus, the final chunk, 01, becomes 010000, which corresponds to Q in the Base64 alphabet. Then, we add one padding character (typically =) for every two bits we had to add for the string to only be made of full chunks. We add two padding characters because four (the number of bits we had to add) divided by two is two.
Thus, you would think that the padded string would be StI/Q==, right? Well, no, because that extra character adds another six bits, meaning that there are still two bits at the end. We can only stay in such a loop two times, so we need to add more zeroes. That means that the final letter is 000000, which is A.
In an earlier post I made, I discussed how regular expressions could be used. Now, I will show you how to implement them in your own C++ program.
The regex Library
The regular expression library was added in C++ 11, so you should have support for it by now. We can start with some basic boilerplate, importing our regular expression library (along with some other libraries that will make our lives easier) and the standard IO library.
In C++, regular expressions must be compiled before they are used. When I say I am going to be passing a regex string or a regex to a function, I am actually going to be talking about this compiled expression. All regexes must be compiled before use.
It is actually very easy to compile regexes, and below, we are compiling regex <html>.+</html> and assigning the compiled expression to a variable called re.
Then, we will do the same thing I mentioned above but naming the expression reg. The reason I am doing this twice is because I want to show the two methods you can use, assigning by value or assigning using the regex class’s constructor.
Determining if a Regular Expression Matches an Entire String
The regex_match function will determine whether an entire string is matched by a certain regex. For example, if we pass the regex hi to it and match it with the string hi, the function will return true, as the regular expression provided matches the entire target string of hi.
However, if we kept the regex the same but changed the target string to shi, the function would return false because while shi contains hi, the regex hi does not match the entirety of shi.
Let’s use an example. I have given one below.
C++
#include<iostream>#include<string>#include<vector>#include<regex>usingnamespacestd;intmain(){ string reStr; cout<<"Enter a regular expression to use the regex_match function on:\n>"; cin>>reStr; string target; cout<<"Enter a target string to use the regex_match function on:\n>"; cin>>target; regex reCompiled =regex(reStr); // Compiling our regex // Actual matching processif(regex_match(target,reCompiled)){ cout<<"\nRegex Matched Entirely!\n";return0;}else{ cout<<"\nRegex Did Not Match Entirely!\n";}return-1;}
A Quick Note: Capturing Groups
Capturing groups in regexes are denoted by parenthesis and are often returned as lists. To make things simpler, let’s use the regex (sub)(mar)(ine). Here we can see that sub, mar, and ine each have their own capturing groups.
Now if we were to use this on the text submarinesubmarine, the regex would match on both submarines separately, so we would get two matches.
Let’s take a closer look at the matches.
These matches would end up having three submatches each due to these capturing groups. If we were to visualize this in hierarchy, we would get the following:
In C++, matching with capturing groups is represented as a list of matches containing lists of each capturing group for each match. For example, if we wanted to get match one, capturing group one, of a list of matches (you will learn about the smatches type in the next section), we would use the code below:
C++
string m1c1 = matches[0][0];
A Quick Note: The smatches Type
The smatches type is used for storing a list of strings as regex matches. It is sort of like a vector, but the shape is fixed to either vector<string> without capturing groups, or vector<vector<string>> with capturing groups.
Determining if a Regular Expression Matches any Substrings
Remember how above, I said that the regex_match function only tells you whether the entire string is matched by a regex? Well, if we want to include substrings, it can get a little more complicated (this is coming from someone with a Python background, where we are pampered with the re library).
For this part of the guide, we will be using the regex_search function, which will tell you if
The regex_search function typically takes three to four arguments. Let’s look at the first method of calling it.
For this method, the function takes three parameters and outputs a Boolean. The parameters are below.
Target (std::string) – This is the string you want to match the regex against
Match Results (std::smatch) – This is the variable of type smatch that will store match results. We will not be using it in this example
Regex (std::basic_regex) – This is the compiled regex that the target is being matched against
The function will return true if any substring of the target string matches the regex, and false otherwise.
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ string s="this variable is called s"; smatch m; regex e =regex("s");if(regex_search(s,m,e) /* Will return true */){ cout<<"Matched! (but not always the entire string)"<<endl;}return0;}
We can also call regex_search using another method, whose parameters are listed below. In this method, we are not only telling the user whether the program is
String Begin and String End – Tells the function to only search the substring in between the string beginning and string ending
Match Results – This is the smatch that will store match results
Regex – The compiled regex that will be used to match against the target string
The function will return true using the same conditions I stated in the previous method, but here what we care about is the fact that the match results are being stored.
The code below will print the first match of the regex, check if there are any matches other than the one it returned, and print the first capturing group. It will keep doing this until there are no other matches. I highly recommend you read the comments in the code below for a better understanding of what it’s doing.
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ string target ="submarine submarine submarine"; regex re =regex("(sub)(mar)(ine)"); smatch m;string::const_iterator searchFrom =string::const_iterator(target.cbegin()); // Begin iteratingwhile(regex_search(searchFrom,target.cend(),m,re)){ // We don't want to keep returning the same match every time, so the code below will exclude this match from the future iterations searchFrom =m.suffix().first; // It is important to know that m[0] would return the entire string ("submarine"), so m[1] will return the first capturing group ("sub") cout<<"We have got ourselves a match! \""<<m[1].str() /* First capturing group of match */<<"\"\n";}}
Regular Expression Find and Replace
The regex_replace function will find and replace all sequences that match the regex.
In the example below, we are telling it to replace all words (including the spaces around them) with “and”
We are also giving it three parameters.
Target – The text that will be replaced accordingly
Regex – The compiled regular expression that will be used on the target
Replace With – The text to replace the matches of the regex with against the target
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ regex re("([^ ]+)"); // Matches every word cout<<"ORIGINAL: this is text\n"; cout<<regex_replace("this is text",re,"and"); // prints "and and and"return0;}
You can also use formatters to incorporate exactly what was replaced using the table below.
Formatter
Example
Explanation
$number(where “number” is replaced by any positive number less than 100)
$2
Replaced with the match of the numberth capturing sequence that triggered the replace (starting from 1, such that $1 will get the first capturing group, not $0) at runtime
Example: Replacing regex matches of “(sub)(.+)” with “2nd CG: $2” using a target string of “submarine” will yield a result of “2nd CG: marine”
$&
$&
A copy of the entire original string, regardless of capturing groups.
Example: Replacing regex matches of “(sub)(.+)” with “String: $&” using the same target string above will result in “String: submarine”
$`
$`
Replaced with whatever came before the match at runtime
Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$`” will get replaced with “a “
$’
$’
Replaced with whatever came after the match at runtime
Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$’” will get replaced with “marine goes underwater”
$$
$$
I wouldn’t call it a formatter exactly; it’s more of an escape sequence. Used when you don’t want the compiler to mistake the literal character “$” with a formatter.
Used when you want to literally type “$” as the text to replace, type “$$”
For example, the code below will put the letters “t” and “e” in parenthesis.
C++
// regex_replace example#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ regex re("([te])"); // Matches either "t" or "e" cout<<"ORIGINAL: thetechmaker.com\n"; cout<<regex_replace("thetechmaker.com",re,"($&)"); // Prints "(t)h(e)(t)(e)chmak(e)r.com"return0;}
In this guide, I will show you how to use the BeautifulSoup library to make a simple program that notifies you when a product on an online site drops in price.
This library runs in the background, scraping static online e-commerce sites of your choice and notifying you when a product drops in price.
Prerequisites
This guide assumes that you have Python installed, pip added to your system’s PATH, along with a basic understanding of Python and HTML.
Installing Required Components
First, let’s install BeautifulSoup and Requests. The Requests library retrieves our data, but the BeautifulSoup library actually analyzes our data.
We can install those two required components by running the command below:
BAT (Batchfile)
pip install beautifulsoup4 requests
Note that depending on what your system’s setup is, you might need to use pip3 instead of pip.
Grabbing Our Sample: Price
In this step, we will be telling BeautifulSoup what exactly to scrape. In this case, it’s the price. But we need to tell BeautifulSoup where the price is on the website.
To do this, navigate to the product you want to scrape. For this guide, I will be scraping an AV channel receiver I found on Amazon.
Then, use your browser’s DevTools and navigate to the price. However, make sure that you have a very “unique” element selected. This is an element that shows the product’s price but is also very specifically identified within the HTML document. Ideally, choose an element with an id attribute, as there cannot be two elements with the same HTML ID. Try to get as much “uniqueness” as you can because this will make the parsing easier.
The elements I have selected above are not the most “unique” but are the closest we can get as they have lots of classes that I can safely assume not many other elements have all of.
We also want to ensure that our web scraper stays as consistent as possible with website changes.
If you also don’t have an element that is completely “unique”, then I suggest using the Console tab and JavaScript DOM to see how many other elements have those attributes.
Like, in this case, I am trying to see whether the element I selected is “unique” enough to be selected by its class.
In this case, there is only one other element that I need to worry about, which I think is good enough.
Basic Scraping: Setup
This section will detail the fundamentals of web scraping only. We will add more features as this guide goes on, building upon the code we will write now.
First, we need to import the libraries we will be using.
Python
import requests as rqfrom bs4 import BeautifulSoup
Then, we need to retrieve the content from our product. I will be using this AV receiver as an example.
If the content you want to scrape is locked behind a login screen, chances are you need to provide basic HTTP authentication to the site. Luckily, the Requests library has support for this. If you need authentication, add the auth parameter to the get method above, and make it a tuple that follows the format of ('username','password').
For example, if Amazon required us to use HTTP basic authentication, we would declare our request variable like the one below:
If that authentication type does not work, then the site may be using HTTP Digest authentication.
To authenticate with Digest, you will need to import HTTPDigestAuth from Request’s sub-library, auth. Then it’s as simple as passing that object into the auth parameter.
Python
from requests.auth import HTTPDigestAuthrequest = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/",auth=HTTPDigestAuth("replaceWithUsername","replaceWithPwd"))
If the content you want to scrape requires a login other than basic HTTP authentication or Digest authentication, consult this guide for other types of authentications.
Amazon does not require any authentication, so our code will work providing none.
Now, we need to create a BeautifulSoup object and pass in our website’s response to the object.
When you use the Requests library to print a response to the console, you generally will want to use request.text. However, since we don’t need to worry about decoding the response into printable text, it is considered better practice to return the raw bytes with request.content.
Basic Scraping: Searching Elements
Now we can get to the fun part! We will find the price element using our sample we got earlier.
I will cover two of the most common scenarios, one where you need to find the price based on its element’s ID – the simplest, or one where you need to find the price based on class names and sub-elements – a little more complicated but not too difficult, assuming you have a “unique” enough element.
If we wanted to refer to an element based on its ID with BeautifulSoup, you would use the find method. For example, if we wanted to store the element with the ID of pricevalue within a variable called priceElement, we would invoke find() with the argument of id set to the value "pricevalue".
Python
priceElement = parser.find(id="pricevalue")
We can even print our element to the console!
Python
print(priceElement.prettify())
Expected Output (may vary)
<divid="pricevalue"><p>$19.99</p></div>
The function prettify is used to reformat (“pretty-print”) the output. It is used when you want to be able to visualize the data, as it results in better-looking output to the console.
Now we get to the tougher part – making references to element(s) based on one or more class names. This is the method you will need to use for most major e-commerce sites like Amazon or Ebay.
This time, we will be using the find_all function. It is used only in situations where it is theoretically possible to get multiple outputs, like when we have multiple classes as the function gives the output as a list of strings, not a single string.
If you are not sure, know that you can use find_all even when the query you give it only returns one result, you will just get a one item list.
The code below will return any elements with the classes of priceToPayorbig-text.
The select function is just like that of the find function except instead of directly specifying attributes using its function parameters, you simply pass in a CSS selector and get a list of matching element(s) back.
The code above selects all elements with the class of both price-value and main-color. Although many use the find or find_all functions, I prefer select as I am already familiar with CSS selectors.
If, and this is not much of a good idea when finding elements, we would like to filter by element type, we will just call find_all with a single positional argument, the element’s type. So, parser.find_all("p") will return a list of every single paragraph (“p“) element.
An element type is one of the broadest filters you can pass into the find_all function, so this only becomes useful when you combine it with another narrower filter, such as an id or class.
Python
parser.find_all("h1",id="title")
That would return all h1 elements with an ID of title. But since each element needs to have its own unique ID, we can just use the find function. Let’s do something more realistic.
Python
parser.find_all("h1",class_="bigText")
This code would return all h1 elements that had a class of bigText.
Below are a few reviews of what we know so far and some other, rarer methods of element finding.
Python
"""Never recommended, but returns a list of ALL the elements that have type 'p'"""typeMatch = parser.find_all("p")"""Finds element with the ID of 'priceValue' using a CSS selector"""idSelMatch = parser.select("#priceValue")"""Finds element with the ID of 'priceValue', except with the BeautifulSoup-native find function and not with a CSS selector"""idMatch = parser.find(id="priceValue")# Same as above"""Extremely rare, but returns a list of elements containing an ID of 'priceValue' OR 'price'"""orIdMatch = parser.find_all(id=["priceValue","price"])"""Returns a list of elements that have the class 'price' OR 'dollarsToPay'. I do not know of a CSS selector that does the same"""orClassMatch = parser.find_all(class_=['price','dollarsToPay'])"""Returns a list of elements that have the class 'price' AND 'dollarsToPay'. I do not know of a find_all argument that does the same"""andClassMatch = parser.select(".priceValue.dollarsToPay")"""Returns the element that has a class of 'v' INSIDE the element of class 't'. This can also be done with ID attributes, but this function only works when the first function is .find(...) or when you are grabbing an element by index after calling .find_all(...). Because .find(...) only returns one element, it will only be returning the first instance of that class name. The code below return the same thing, however 'inMatch3' returns a list"""inMatch = parser.find(class_="t").find(class_="v")# Most basic way to do itinMatch2 = parser.find_all(class_="t")[0].find_all(class_="v")[0]# Because .find_all(...) works on the final element, the '[0]' is unnecessary, we just do it so we don't get a one-element listinMatch3 = parser.find_all(class_="t")[0].find_all(class_="v")# Returns a one-element list
Now that we know how to search elements, we can finally implement this in our price drop notifier!
Let’s see if our request is successful. We will be printing out the entire file to check.
Python
print(parser.find("html").prettify())
And we are not.
Hmmm, so we have to bypass Amazon’s CAPTCHA somehow, so let’s try adding headers that mimic a normal browser!
I will be adding headers to rq.get(). Make sure to replace my AV channel receiver link with the product you want to scrape.
Firstly, it is important to know that Selenium has its own methods for finding elements in a HTML document, but for the sake of this guide, we will just be passing the source code of the website to our parser.
Think of Selenium as a browser running in the background with some selection abilities. Instead of sending the requests to the website by crafting our own headers, we can use Selenium to spin up an invisible browser that crafts the headers for us. We should no longer get a CAPTCHA screen because Amazon shouldn’t be suspicious that a robot is browsing the page – we are technically using a legitimate browser, but with parsing capabilities.
Installation of Selenium can be done with the command below. We will also be installing win10toast so you get a proper toast notification whenever a price drop is detected.
BAT (Batchfile)
pip install seleniumpip install win10toast
If you are looking for how you can uninstall Requests because you don’t need it anymore, think twice because Selenium depends on Requests anyways.
Now, clear your entire Python file because we are going to need to do a short and quick rewrite of our code to use Selenium.
Like always, we will start by importing the required modules. Make sure you replace chrome with the name of a browser you have installed on your system, preferably the most resource efficient one.
Python
from selenium import webdriverfrom bs4 import BeautifulSoupfrom selenium.webdriver.chrome.options import Options # Imports the module we will use to change the settings for our browserimport time # This is what we will use to set delays so we don't use too many system resourcesfrom win10toast import ToastNotifier # This is what we will use to notify if a price drop occurs.notifier =ToastNotifier()# Assign our notifier class to a variable
Then, we will need to set some preferences for the browser we are about to start. Let’s start by declaring an Options class and using it to make the browser invisible or run it in “headless” mode. While the arguments below are for specific browsers, I would just execute them all because I have not tested each argument individually.
Python
browserOptions =Options()browserOptions.headless=True# Makes Firefox run headlessbrowserOptions.add_argument("--headless=new")# Makes newer versions of Chrome run headlessbrowserOptions.add_argument("--headless")# Makes older versions of Chrome run headlessbrowserOptions.add_argument("--log-level=3")# Only log fatal errors
Now, we will initiate the browser in the background. Again, make sure you replace Chrome with whichever browser you want to use for this project.
Then, we can use what we already know about BeautifulSoup to grab the price of our element. Remember to replace the code below with one tailored to your sample.
Next, let’s strip the $ symbol from the price and convert it into a floating-point decimal.
Python
price =float(price.strip("$"))
Then, we can set a variable to compare with the current price.
Python
previousPrice = price
Now, we loop infinitely to see whether the price changed.
Python
whileTrue:
Insert a new line and then indent the code we will write from this point forward.
Now, every two minutes (120 seconds), we refresh the page and compare the price we just got to our previous price.
Python (place each line indented inside while loop)
browser.refresh()# Refreshes the browser# Now that we may have a new price, we have to redfine our parser and price variables to adapt to that new page codeparser =BeautifulSoup(browser.page_source,"html.parser")price = parser.select(".a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay")[0].find_all(class_="a-offscreen")[0].textprice =float(price.strip("$"))# Next, we compare the two prices. If we find one, we alert the user and update our price threshold. We will also be looking for price increases.if(price<previousPrice):print(f"Price DECREASED from ${previousPrice} to ${price}!") notifier.show_toast("Price Drop!",f"The price decreased from ${previousPrice} to ${price}!")elif(price>previousPrice):print(f"Price INCREASED from ${previousPrice} to ${price}!") notifier.show_toast(":(",f"The price increased from ${previousPrice} to ${price} :(")# Now, we can tell the user we refreshedprint(f"Refreshed! Previous Price: ${previousPrice}, and new price ${price}")previousPrice = price# And then we wait for two minutestime.sleep(120)
And just like that, you are finished! I hoped this project was useful to you!
Regular expressions (abbreviated as a “regex”) are useful tools that help easily find and match text in strings and files. They sort of function like the typical find-in-document feature you are probably used to, only more advanced. However, they can definitely be too advanced for an average person to comprehend. In this guide, I will easily explain this widely used and useful feature so you can search files like a pro.
Something Worth Noting
It is worth noting that regexes are very useful and have lots of features to utilize, but can also be very non-standardized. There are always features in other regex parsing engines that are not supported by others. The features I will be displaying in this guide are the most widely supported ones, but there are always going to be features that I won’t teach you how to use here either because they are not known or because they are defunct and not considered good practice.
Basic Matching
Regexes can match specific strings, no complex syntax needed. For example, to match the text foo in notnotnotfoonotnot, your regex would simply be foo.
This works with multiple instances too; the regex foo would match both instances of foo in notnotnotfoonotnotfoo.
Something else that you should know about basic matching is that there will be two seperate matches if the same matching string is used multiple times (together or not).
This means that the regex foo would match all three foo instances in notnotfoofoonotfoo (notice how there are separate matches for the two instances of foo, even though said instances are together).
Match Any Character
The period selects any character. Note that each character is matched separately.
For example, the regex . in the text foo will match f, o, and the second o.
Notice how in the image above, even special characters and numbers are matched.
Letter Range Match
Let’s say in the text abcdefghijklmnopqrstuvwxyz, we want to match just e, f, g, h, and i. To do this, we can select the letter range that we want (in this case, e through i) using the regex [e-i].
Note that these letter ranges do not have to be subsequent; we can have the regex [a-c] in the text azaczbca match a, a, c, b, c, and a.
Number Range Match
Let’s say in the text 019201836y7 you only want to select the numbers 0, 1, 2, the second 0, and the second 1. In other words, we want to select any number from zero to two, including themselves. We can do this using the number range match, and in this case, we can use the regex [0-2].
These letter ranges also don’t have to be consecutive.
Character Set
Now let’s say you want to match specific characters that can’t be matched with a set. For this, we will use a character set. For example, we can match car and bar in the text car,bar using a character set. Because only the beginning letters of each word change, we can use a character set to match the c and b, and then use basic matching to match the ars. To do this, we can put in the characters we want to put in the set in square brackets.
Ignored Character Set
How about if we wanted to match every character except for c and b. To do this we can use an ignored character set to get the job done. If we wanted to match only tar in the text car,bar,tar, we can use an ignored character set to ignore c and b, then use basic matching as all three words have different initials but the same of everything else. To do this, we can put the characters we want to ignore in square brackets with the character ^ coming before them.
The Asterisk
The asterisk (*), when placed outside a character indicates that said character should be matched even if it does not occur at all or occurs at least once side by side. For example, the regex a* will match at least zero of a, so aaaa will be matched in the text aaaab. This is different from basic matching, as the regex a will count every a as a separate match, while a* will keep all the as in one match as many times as needed to complete the match.
Another examples is that the asterisk is like saying the character that comes before it is optional or can occur multiple times in a row. The regex ab*c in the text abc,abbc,ac will match abc, abbc, and ac.
The Question Mark
The question mark is just like the asterisk in the sense that the character that comes before it is deemed optional, but different in the sense that it will not match characters that occur multiple times in a row. For example, the regex ab?c will match abc and ac, but not abbc.
The Plus Symbol
The plus symbol is also just like the asterisk in the sense that the character that comes before it can occur multiple times in a row, but different in the sense that it does not indicate a character is optional. If we go back to our abc example, we can see that the regex ab+c in abc,abbc,ac will match abc and abbc, but not ac.
Conclusion
And that’s it – you’re done! There are so many other regex sequences you can use, and they just won’t fit into a beginner’s guide. As you saw above, regular expressions are very useful tools, and there are many ways to apply this to your needs.
In this guide, I will be showing you how to install Arch Linux from scratch. You can do this on a physical machine, but I will be doing it on a virtual machine.
What is Arch Linux?
Arch Linux is a minimal Linux distro that is meant for power users and advanced programmers. It does not come with a built-in installer, so we will have to install it manually.
Downloading the ISO
First, head to the Arch Linux downloads page, scroll down until you find the download mirrors, and then choose a link to download, preferably from your country so you get the fastest download speed.
VM Configuration
If you are installing Arch Linux on physical hardware and you are not using a VM to install Arch, skip this section.
If you are on VirtualBox, there should be an Arch Linux preset. If you are on VMware, select Other Linux 5.x Kernel (64-bit).
Giving your VM 8 GB of RAM is a lot more than needed, but if you are going to be using Arch for power-intensive tasks and don’t mind the VM taking up all your host’s RAM, go for whatever fits for you as long as it meets the system requirements of 512 MBminimum, but 2 GB recommended for streamlined daily use.
Now, give your VM any amount of storage you feel fitting, but make sure it meets the system requirements of 1 GBminimum, but 20 GBrecommended.
If you want things to go a little faster on your Arch Linux VM, giving it two processors is recommended. One processor should be enough, though.
If you plan on installing a desktop environment, enable 3D acceleration and give a reasonable amount of VRAM to the guest OS.
And lastly, make sure the CD drive is set to read as the ISO you just downloaded.
Making a Bootable USB
If you are using a VM, skip this section.
Use a tool like Rufus to flash the Arch Linux ISO to a flash drive. Then plug it into the system you want to install Arch Linux to. This guide does not cover dual-booting and assumes you do not have an existing OS installed on your system.
Boot Priority Configuration
Make sure the hard disk is the first on the boot priority list. This is not required; it just makes the final step of the installation a lot quicker.
You can do this in the VM’s settings, the UEFI firmware settings, or the BIOS interface.
Starting the System
If you are on a VM, start it up. If you are on hardware, plug in the thumb drive that you flashed the ISO to and start the machine.
You should see an Arch Linux splash screen. Whichever entry comes first in the list is likely the one that boots to Arch (the name of the entry changes, but typically goes along the lines of “Boot Arch Linux (x86_64)”). Select Boot Arch Linux, or the entry that does so, and click Enter.
After running some tests, you should be dropped into a root Linux shell. Do not remove your thumb drive or installation media, as we have not installed Arch yet and need to do so using the thumb drive.
Accessing the Internet
Let’s check if we have connection to the internet by running the command below (“google.com” can be replaced with a website of your choice):
Bash
pinggoogle.com
You should see packets return. If you do, this means you have online access, which is a necessity for this installation. You can hit Control + C to stop pinging the website.
If you are on a Virtual Machine with no internet, try making sure that you have host internet access, and then try enabling the network adapter in the VMware or VirtualBox settings.
If you are on a physical machine without internet, try using an Ethernet cable, but below you will know how to use iwctl to connect to a Wi-Fi network.
Using iwctl to Connect to a Wi-Fi Network
This part of the guide should only be followed if you do not have Internet access and want to use a wireless internet connection instead of a wired one.
Run the command below to open the iNet Wireless Daemon CTL.
Bash
iwctl
Expected Output
[iwd]#
Now, you can list the wireless adapters using the command below.
Bash
devicelist
You should see a device called wlan0. It is best to go forward using that one, but you can select another wireless adapter if you know what you are doing. If you do not have that device, then you do not have a wireless adapter plugged into the computer, the device is connected but under a different name (which is unlikely), or the computer does not recognize it.
Use the command below to list all the wireless networks found using that device. You can replace wlan0 with the adapter you chose earlier.
Bash
stationwlan0get-networks
You should see a list of networks. Take note of the network’s SSID you want to connect to. Then, you can run the command below to connect to the wireless network, replacing “WirelessNet” with the SSID of your wireless network and wlan0 with the wireless adapter you want to connect to the network using.
Bash
stationwlan0connectWirelessNet
After typing in the Wi-Fi password (if needed), you may now connect to the network. You can test your connection by using ping google.com and waiting for packets to return. If none return, then you might have done something incorrectly when setting up the network.
Setting NTP Time
Now that we have our network up and running, we can enable network time using the command below:
Bash
timedatectlset-ntptrue
Partitioning
Now comes the tricky part. We have to partition our drives manually, so make sure to follow my steps carefully.
Run the below command to get a summary of all the partitions on your drive.
Bash
lsblk
Make sure you choose the right disk to partition, as choosing the wrong one will destroy all of your data. Run the below command to set up partitioning, replacing /dev/sda with the name of the disk you want to format and install Arch Linux on.
As for the label type, it depends what your needs are. If you are installing to a new physical system with a disk size larger than 2TB, select gpt and hit Enter. If you either don’t have a physical system to install Arch to or are installing Arch to a disk smaller than 2TB, use dos. Now, at the menu with drives listed, select the free space and click New. When asked for your partition size, enter the amount needed for your bootloader. If you will be using the GRUB bootloader, enter 128M and hit Enter. If not, specify the amount needed for your bootloader.
Now, select the newly created partition and hit B to make the selected drive bootable.
Select the free space, and click New. The size should automatically default to the remaining storage on your drive. Make the partition fill up the rest of the drive, and click Enter to create the partition. You should not make this partition bootable.
Many people prefer creating home and swap partitions but these are mostly considered redundant nowdays.
Select Write and click Enter, then select Quit and click Enter.
There is only one problem left to solve now – the drives are not in the format of ext4. To solve this, run the commands below, replacing /dev/sda1 and /dev/sda2 with the names your newly created boot and OS partitions.
Bash
mkfs.ext4/dev/sda1mkfs.ext4/dev/sda2
Mounting our Partitions
Now that we are done with arguably the hardest part of Arch installation, we need to mount our drives, which is where the preparation ends and the actual installation starts.
To begin, lets mount our root partition (the partition that is not bootable, /dev/sda2 in my case) to a mount point (this can be anything you want, but traditionally this has always been /mnt. I will be using /mnt, as I do not see any reason to stray from tradition in my case). We can do this using the command below.
Bash
mount/dev/sda2/mnt
In our mount point, let’s create a folder called boot to mount our other drive to.
Bash
mkdir/mnt/boot
Now, let’s mount our boot partition (the one that we flagged bootable earlier, /dev/sda1 in my case) to the folder we just created.
Bash
mount/dev/sda1/mnt/boot
To see if we did everything correctly, we can run the command below.
Bash
lsblk
In the output, you should be able to see partitions under the drive /dev/sda and their respective mount points.
Installing Arch Using pacstrap
Now, we can begin installing system files to Arch.
We can use pacstrap to install Arch Linux and some other packages we want pacstrap to bundle in with our Arch installation. Replace /mnt with the mount point you mounted your root drive to, and vim with some other text editor that you prefer and some other pre-installed packages you want on Arch.
Bash
pacstrap/mntbasebase-devellinuxlinux-firmwarevim
Below ae explanations of some of the packages:
Base: This package contains basic tools that you would want no matter which Linux distribution you are installing.
Base-Devel: This contains developer tools such as compilers, which are needed for some Linux components.
Linux: This is the core Linux kernel that runs everything behind the scenes.
Linux-Firmware: This contains firmware that makes Arch compatible with common hardware
Vim: This can be replaced with any text editor. There is some text editing we are going to have to do, so we need a text editor.
Once pacstrap exits, we can generate an fstab file that lists all the drives Linux could try booting from.
Generating our FSTab File
This is extremely easy to do. Run the command below, replacing /mnt with the mount point you specified earlier.
Bash
genfstab-U/mnt>>/mnt/etc/fstab
What the command should do is write a FSTab file to the drives. The -U flag makes the file reference drives by UUID instead of drive name. This is good because drive names may change in the future, which could mess up your boot configuration if you don’t reference drives by UUID, as UUID never changes.
Jumping to our Arch Installation
It is finally time to change from our installation medium to our disk. Do not remove your installation medium until the end of this guide. You might need it if something breaks.
Bash
arch-chroot/mnt/bin/bash
After this command, do not reboot yet. We still have to install our boot manager.
Installing Basic Components
Now that we are in Arch, we have access to the Pacman package manager. We can use it to install basic components like a network manager for accessing the internet, and a boot manager so we can boot into the system. I will be installing GRUB as a boot manager, but you can install something else.
Use the command below to install these components.
Bash
pacman-Snetworkmanagergrub
Configuring the Network Manager to Run on Startup
If we want internet at boot, we are going to have to enable NetworkManager’s system service. we can do this using the command below.
Bash
systemctlenableNetworkManager
Configuring the GRUB Boot Manager
Now, we have to configure what our system boots using. When we ran the Pacman command, we downloaded GRUB, but we did not install or configure it. Let’s install GRUB using the command below, replacing /dev/sda with whatever drive you are using to install Arch. We are not going to be using /dev/sda1 or /dev/sda2 because it is critical that you install it to the drive, not the drive partition.
Bash
grub-install/dev/sda
Now, we can make a GRUB configuration file using the command below.
Bash
grub-mkconfig-o/boot/grub/grub.cfg
Take a look at the output and make sure that it says both an initrd image and a Linux image were found. If it does not find these images, the most likely cause is incorrectly installing the Linux kernel using pacstrap.
Setting a Password
Now, run the command below to create a password for your root user.
Bash
passwd
Configuring Locales and Timezones
Use the command below to enter the locale configuration file, replacing vim with whatever text editor you installed earlier.
Bash
vim/etc/locale.gen
Now, use the arrow keys or the scroll wheel to scroll down to the locale you want to use. I will be using United States English, so I will scroll down to en_US and uncomment (remove the # before) both the UTF and ISO formats. If you are using Vim, you might have to hit the I key on your keyboard before you can actually type anything.
Write the file and exit your text editor. To write and exit Vim, we can hit the Escape key on our keyboard and type :wq.
Now that we have our locales configured, we have to apply the changes by generating the locales. We can do this using the command below.
Bash
locale-gen
Now, we also have to create a file called locale.conf to define our language in. Use the command below, once again replacing vim with your desired text editor.
Bash
vim/etc/locale.conf
In the file, type LANG=en-US.UTF-8, once again replacing en-US with whatever locale you are using. Exit Vim.
Now that we have the timezones prepared, we have to link them to make our system clock show the right timezone. Type ln -sf /usr/share/zoneinfo and click Tab. This will list all the broadest timezones. Keep making the directories more specific, hitting Tab to see the available options every time, and after you are done typing a city, hit Space and type /etc/localtime.
Setting Our Hostname
Now, we can set our hostname. This is the name that the Arch machine will use to communicate with other devices over your network. By default, your hostname is archiso. If you are happy with that and don’t want to change it, you can skip this section.
Use your prefered text editor to create /etc/hostname. Do not include a file extension. Type whatever you want your system hostname to be, and exit.
Unmounting Our Disk
Now, we can exit our chroot jail by using the command below.
Bash
exit
Now would be a good time to check and make sure your hard drive is first boot priority. Make sure that when you return, you are in your installation medium and not in the actual Arch installation.
Unmount our Arch installation with the command below, replacing /mnt with the mount point you specified earlier.
Bash
umount-R/mnt
Now, we can boot into our new installation of Arch Linux using the command below. Once you have booted in, you may remove the installation medium.
This guide is a part two to a previous guide I made, called The Simple Guide to AI and Machine Learning With Python. This guide is simply how you can improve accuracy to the model you made in that guide, meaning that I’m going to assume you have already completed the previous guide before going on to follow this guide.
In the previous guide, we learned how you can use dense neural networks to make a program that recognizes handwriting. Well, that neural network was not exactly very accurate, as it had a tendency to get numbers wrong unless it was specifically modified for those numbers. As you probably know by now, you would probably want the neural network to recognize any number you give it without having to optimize the network for every single number that comes to it.
Convolutional neural networks were made to solve this problem. Rather than training off of the overall image, convolutional neural networks recognize tiny features in the image and learns those. For example, rather than focusing on the entire image of a hand-drawn three, the network will learn that a three has two curves that are stacked vertically, which will help it recognize any other threes in the future, no matter how it was drawn or whether the neural network was optimized for the number three.
Step One: Initial Setup
For this step, we can just use the code that we used in the previous tutorial to prepare the MNIST dataset.
Python
import tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras.datasets import mnistfrom tensorflow.keras import backend as Kimport numpy as npimport matplotlib.pyplotas plt%matplotlib inline# helper functionsdefshow_min_max(array,i): random_image = array[i]print("min and max value in image: ", random_image.min(), random_image.max())defplot_image(array,i,labels): plt.imshow(np.squeeze(array[i])) plt.title(" Digit "+str(labels[i])) plt.xticks([]) plt.yticks([]) plt.show()defpredict_image(model,x): x = x.astype('float32') x = x /255.0 x = np.expand_dims(x,axis=0) image_predict = model.predict(x,verbose=0)print("Predicted Label: ", np.argmax(image_predict)) plt.imshow(np.squeeze(x)) plt.xticks([]) plt.yticks([]) plt.show()return image_predictimg_rows, img_cols =28,28num_classes =10(train_images, train_labels),(test_images, test_labels)= mnist.load_data()(train_images_backup, train_labels_backup),(test_images_backup, test_labels_backup)= mnist.load_data()print(train_images.shape)print(test_images.shape)train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols,1)test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols,1)input_shape =(img_rows, img_cols,1)train_images = train_images.astype('float32')test_images = test_images.astype('float32')train_images /=255test_images /=255train_labels = keras.utils.to_categorical(train_labels, num_classes)test_labels = keras.utils.to_categorical(test_labels, num_classes)print(train_images[1232].shape)
Expected Output
(60000, 28, 28)(10000, 28, 28)(28, 28, 1)
Now that we have already put in the initial setup of our code, we can jump straight to creating our network.
Creating Our Network
Similar to what we did with the densely connected network, we are still going to have epochs, or the amount of times the network goes through the entire set over again.
With that explanation out of the way, we can define our model.
Python
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropoutepochs =10model =Sequential()
Now, let’s start adding the layers of our neural network.
Explaining Convolutional Layers
With our previous network, we added three dense (fully connected) layers. With our new network that uses convolutional neural networks, the layers work differently.
Convolutional layers consist of groups of neurons called filters that move across the image and activate based on the pixels they read. Those groups will then learn how to recognize features in the data.
It is possible to adjust amount and size for filters in your neural network, which we will change to our liking. Bigger filters can observe larger parts of the image at once, while smaller filters gather finer details about the image. A higher amount of filters means that the neural network can recognize a wider range of image features.
There are many advantages of having layers and filters work this way. For one thing, smaller filters can be more computationally efficient by only examining a small part of the image at once. Furthermore, as filters are moved across the entire image, the neural network will not be affected by feature displacement (occurs when a feature is common to two images, but in different spots of an image). Just like reality, filters focus on a small area of the image, so they are not distracted by the other parts of an image.
We will be using multiple convolutional layers to complete our new-and-improved handwriting recognition software.
Implementing Convolutional Layers
When we use Keras, we can easily take advantage of its functionality to easily create convolutional layers that we will then use in our model. We will use the Conv2D function to create the first layer of out neural network.
In the case below, we will have 32 filters, a kernel size of (3,3), an input shape – which we saved to the input_shape variable when we ran the setup code at the beginning – of (28,28,1), and an activation function of ReLU. I go more in-depth into what ReLU is in my previous guide.
The Conv2D function creates 2D convolutional layers, meaning that they scan across flat data, like images.
Explaining Pooling Layers
When you use convolutional layers, things can get quite computationally intensive, which is where pooling layers come in. Increasing the number of neurons will increase the number of computation time required. Pooling layers are essentially filters that move in specified strides across the image, simplifying each of the filters’ contents into a single value. This, based on the size and stride of the filter, shrinks the output image.
For this scenario, we will have a 2×2 filter with a size of 2. This halves the image’s row and column count, simplifying the data without too much loss of specificity.
Python
model.add(MaxPooling2D(pool_size=(2,2)))
Most networks have at least one set of alternating convolutional and pooling layers.
More Convolutional Layers
Convolutional layers are designed to examine the low-level features of an image. If we add more, we may be able to start working with higher-level features.
We define the layer the same way we defined the previous one, but now we have 64 filters, not 32. We also do not specify the input shape, as it is inferred from the previous layer.
Dropout layers are layers that take a percentage of all input neurons and deactivate them randomly. This forces other neurons to adapt to the task. When larger and more complicated networks lack a dropout layer, the network risks being too dependent on a single set of neurons rather than all neurons learning. This is called Overfitting and can change your network output for the worse.
Below, we will have our dropout layer take 30%, or 0.3 neurons to deactivate randomly.
Python
model.add(Dropout(rate=0.3))
Dense and Flatten Layers
After all the convolutional and pooling layers, we will need a layer to help make our final decision. This will be a regular, fully connected dense layer. Before we connect this layer, we will need to flatten the image’s filters.
We can start by flattening the image using the Keras Flatten layer.
Python
model.add(Flatten())
Now, we can add a dense layer with ReLU activation and 32 neurons.
Similar to the fully connected neural network we made in the previous guide, we will need a layer to shrink the previous dense layer down to just the number of classes. Also similar to before, the final output is decided by using the class with the highest weight.
Below, we will add a dense layer to be our output layer. The number of neurons should be 10 because there are ten possible output classes, and the activation should use Softmax.
Python
model.add(Dense(units=10,activation='softmax'))
Model Summary
Now, we can print out our model summary:
Python
model.summary()
Expected Output (Lines Providing no Useful Data are Blurred)
Now we will compile the network. The loss and metric will be the same as the ones that we use in the previous guide, Categorical Cross Entropy and accuracy respectively. However, we will use RMSProp (Root Mean Squared Propagation) as our training algorithm. RMSProp is one of many training algorithms that Keras can use to teach the network how to actually improve, optimizing the loss to make it as small as possible. We will achieve this using RMSProp.
The fit function is the one that actually does the training.
Now we can look at the parameters of the training function.
train_images and train_labels state the data that this neural network model will be trained on. The images are the pieces of data given to the network, and the network tries to find out the appropriate label
batch_size allows us to put the network’s data into batches. We can always change it later, but for now we have set it to 64
epochs defines the number of epochs (times the network reiterates on the training data) the network should use
validation_data defines the data the model is testing itself on
We have turned shuffle on so Keras shuffles the training data after every epoch and isn’t relying on the order of the data to train on
Now, we have to test the model on data it hasn’t seen yet. To do this, we will use the evaluate function. Loss and accuracy are percentages returned in decimal format.
