Author: ishanajay2012

Adding Menu Bar Functionality to FeatherText

As you may know, I recently began building my own online text editor. We added a menu bar to it, but not any functionality. Today, I will be adding functionality to my menu bar.

Adding a Context Menu

To begin, I added these lines right after we declare the top menu bar container.

index.html

<div id="top_menu_options">
        <!-- PLACEHOLDER TEXT: -->
        Lorem ipsum dolor sit amet consectetur, adipisicing elit. Qui, nobis accusantium. Blanditiis similique, architecto porro quod consequuntur consectetur, est culpa ad deserunt eveniet, eius laborum! Voluptatum, sapiente? Labore, ut et.
</div>

To give it some style, we can start a new CSS ruleset. Let’s make it float above the other elements and have a minimum width and height of 200 pixels, a preferred width of 20% viewport width, a maximum width of 300 pixels, a maximum height of 500 pixels, and a preferred height of 30% viewport height.

style.css

#top_menu_options {
  z-index: 2;
  position: absolute;
  width: clamp(200px, 20vw, 300px);
  height: clamp(200px, 30vh, 500px);
}

Let’s give it a background blur and a background color. It should change with the theme, so let’s add a variable to our .dark ruleset.

style.css

.dark {
/* We must write it this way so that we can give it a background blue with backdrop-filter: blur(...) and rgba(...) */
  --context-bg-color: 49, 49, 49; /*#313131*/
}

Then we can apply this color (mixed with some transparency) and a background blur to our context menu.

style.css

#top_menu_options {
  backdrop-filter: blur(1.7px);
  background-color: rgba(var(--context-bg-color),0.7);
}

It appears to be a little too below the menu bar for my liking, so I will shift it up a little here.

style.css

#top_menu_options {
  transform: translateY(-3%);
}

We won’t need it visible at all times, so let’s keep it hidden by default.

style.css

#top_menu_options {
  display: none;
}

If our screen isn’t tall enough, we will need to scroll, but the scrollbar does not look good on the page, so we will hide it.

style.css

#top_menu_options {
  overflow-y: scroll;
  scrollbar-width: none;
}

Importing a Script

First, I put this line right after we set the title of the webpage.

index.html

<script src="scripts/top_menu_bar.js" defer></script>

Within this script, I put some starter code in that would allow us to log to the console while telling us where the message was coming from and loop over every menu item to give them click and mouseenter listeners.

top_menu_bar.js

function log(txt) {console.log(`[top_menu_bar.js] ${txt}`);}
log("Script loaded!")

const context_menu = document.getElementById("top_menu_options");
Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {

    top_menu_item.addEventListener("click",()=>{
        
    })
    
    top_menu_item.addEventListener("mouseenter", () => {
    
    })

})

Keeping Track

In the future, it will come in handy to keep track of the context menu state. That is why we are going to add this line right after we declare the log function.

top_menu_bar.js

const context_menu_state = { clicked: "" }; // I can already tell that we will need to keep track of the clicked item, so I'm adding it here now

We should also set the actual context menu itself to a variable.

top_menu_bar.js

const context_menu_container = document.getElementById("top_menu_options");

Because we will be keeping context_menu_state['clicked'] as a string for easy debugging, we should probably link those strings to actual HTML elements we can manipulate in the DOM.

top_menu_bar.js

const menu_bar_items = {}; // We will populate this later. We should be focused on scalability, so using preset menu items would become more tedious in the long run

Then, we can populate this object.

top_menu_bar.js

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => { // We have to convert it to an array to run forEach on it.
    menu_bar_items[top_menu_item.innerText] = top_menu_item;
});

Showing the Context Menu on Click

Let’s get back to our click listener in the loop over every top menu item and show the context menu under it.

top_menu_bar.js

context_menu_container.style.display = "block";
context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";
context_menu_state.clicked = top_menu_item.innerText; // We declared this object earlier, now we need to update it

Let’s also give an indication of the selected item in the menu bar by adding a CSS class to it so we can style it later. This also involves first removing it from any other menu items.

top_menu_bar.js

Object.values(menu_bar_items).forEach(val => {
  val.classList.remove("force_selected_border_underline");
});
top_menu_item.classList.add("force_selected_border_underline");

However, inspecting the top menu items gives us a problem. The mouseenter event does not trigger for HTML margins, so the spaces between the menu items won’t trigger an event. This can be fixed by nesting the text in a p element, like this.

index.html

<div class="top_menu_item">
  <p class="top_menu_item_text">Help</p>
</div>

And applying the border only to the inner p element.

style.css

#top_menu_container p.force_selected_border_underline {border-bottom-color: rgb(var(--accent-1)) !important;}
#top_menu_container p.top_menu_item_text {
    clip-path: polygon(
        0 0,
        100% 0,
        100% calc(100% - 4px),
        0 calc(100% - 4px)
    );
    transition: clip-path 50ms linear;
    transition-delay: 50ms;
    margin: 0;
    padding: 0;
    border-bottom: 6px solid var(--text-color);
}
.top_menu_item {
    cursor: default;
}
#top_menu_container .top_menu_item:hover p.top_menu_item_text, #top_menu_container p.force_selected_border_underline {
    clip-path: polygon(
        0 0,
        100% 0,
        100% 100%,
        0 100%
    );
}

Now, outside of any event handler but within the loop over the top menu items, let’s store this text in a variable.

top_menu_bar.js

const top_menu_item_inner_p = top_menu_item.getElementsByClassName("top_menu_item_text")[0];

At the beginning of the script, we should link the text of the item to the p element within it, so let’s make an object for that

top_menu_bar.js

const menu_bar_inner_p_items = {};

And populating it with the outer menu items.

top_menu_bar.js

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {
    menu_bar_items[top_menu_item.innerText] = top_menu_item;
    menu_bar_inner_p_items[top_menu_item.getElementsByClassName("top_menu_item_text")[0].innerText] = top_menu_item.getElementsByClassName("top_menu_item_text")[0];
});

Then, we need to apply the class change to the inner paragraph element, so let’s modify our click listener within the loop to account for this.

top_menu_bar.js

context_menu_container.style.display = "block";
context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";
context_menu_state.clicked = top_menu_item.innerText;
Object.values(menu_bar_inner_p_items).forEach(val => {
    val.classList.remove("force_selected_border_underline");
});
top_menu_item_inner_p.classList.add("force_selected_border_underline");

By now, our JS should probably look something like this:

top_menu_bar.js

function log(txt) {
    console.log('[top_menu_bar.js] ', txt);
}
log("Script loaded!");

const context_menu_state = { clicked: "" };
const menu_bar_items = {};
const menu_bar_inner_p_items = {};
const context_menu_container = document.getElementById("top_menu_options");

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {
    menu_bar_items[top_menu_item.innerText] = top_menu_item;
    menu_bar_inner_p_items[top_menu_item.getElementsByClassName("top_menu_item_text")[0].innerText] = top_menu_item.getElementsByClassName("top_menu_item_text")[0];
});

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {
    const top_menu_item_inner_p = top_menu_item.getElementsByClassName("top_menu_item_text")[0];

    top_menu_item.addEventListener("click", () => {
        context_menu_container.style.display = "block";
        context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";
        context_menu_state.clicked = top_menu_item.innerText;
        Object.values(menu_bar_inner_p_items).forEach(val => {
            val.classList.remove("force_selected_border_underline");
        });
        top_menu_item_inner_p.classList.add("force_selected_border_underline");
    });

    top_menu_item.addEventListener("mouseenter", () => {
        
    });
});

Allowing the User to Peek Context Menus

In a menu bar, if you click on a menu item and move your mouse over to another item, the options move from the original one you clicked and appear under that item. Let’s recreate this by adding code to the inside of our mouseenter event. I am going to avoid mouseover since that fires multiple times if your cursor is in the element, not just when your cursor enters the bounds of the element.

Within this event, we first need to make sure that something was clicked before we show the context menu.

top_menu_bar.js

if (context_menu_state.clicked !== "") {

}

Then within this if statement, we can make sure that the context menu is visible and then make it appear under the currently hovered menu item.

top_menu_bar.js

context_menu_container.style.display = "block";
context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";

We should also give a visual indication of the currently hovered item, so let’s add a class to it called force_white_border_underline. This class should create a distinction between the currently hovered item and the currently selected item. We should only add it if the selected element actually does differ from the hovered element, though.

top_menu_bar.js

Object.values(menu_bar_inner_p_items).forEach(val => {
    val.classList.remove("force_white_border_underline");
});
if (context_menu_state.clicked !== top_menu_item.innerText) {
    menu_bar_inner_p_items[top_menu_item.innerText].classList.add("force_white_border_underline");
}

Then, we can implement it in CSS by adding it to the ruleset that sets clip-path to show the entire border, which makes that ruleset look like this.

style.css

#top_menu_container .top_menu_item:hover p.top_menu_item_text, #top_menu_container p.force_selected_border_underline, #top_menu_container p.force_white_border_underline {
    clip-path: polygon(
        0 0,
        100% 0,
        100% 100%,
        0 100%
    );
}

Hiding the Menu Options Whenever the User Clicks Elsewhere

We now have a fully functional menu bar system, unless you want to leave the menu bar and exit completely. Sure, we could use focusout, but the menu bar items aren’t even allowed to have focus, and even if we set their tabindex to zero, clicking on a context menu will instantly hide it, since a menu bar item lost focus. On most Windows systems, this is the case, however what if in the future I decide that the context menu options should remain visible until after the action has completed. The goal here is to get the most flexibility.

To accomplish this, we will use a click event on all of the document (since the menu options are outside the scope of the top menu items) and then hide and remove the classes we may have added to it if the click event was not fired because the user clicked on another menu bar item (or its children) or the context menu and its children.

This code to add the event listener will not be added in any scopes, at the every end of the file.

top_menu_bar.js

document.addEventListener("click", (e) => {
    if (!context_menu_container.contains(e.target) && !Object.values(menu_bar_items).some(item => item.contains(e.target))) {
        context_menu_state.clicked = "";
        context_menu_container.style.display = "none";
        Object.values(menu_bar_inner_p_items).forEach(menu_bar_inner_p => {
            menu_bar_inner_p.classList.remove("force_selected_border_underline", "force_white_border_underline");
        });
    }
});

Now, our context menu behavior works flawlessly!

Structuring the Options for the Context Menu

In order to gain more individual control over each menu option, there will be an unordered list within this context menu container, and each list item will be an option of the menu. It will also be a flex container (which will help with responsive layouts and horizontal alignment later). Each list item will have two more flex containers inside of it, one aligned to the left and one to the right (we will use justify-content: space-between; on the list item to accomplish this). For now, the left div will just contain the action while the right will contain the keyboard shortcut or an arrow, if the action is meant to trigger a submenu.

Below is an example of the structure. We can add more list items later.

index.html

<div id="top_menu_options" class="comfy_spacing"> <!-- .comfy-spacing will be useful for customization later -->
        <ul id="top_menu_options_ul" class="undecorated_list">
            <li class="top_menu_options_li flex_container">
                <div class="flex_container">
                    <p class="top_menu_options_choice_inner_p_main_text">New</p>
                </div>
                <div class="flex_container">
                    <p class="top_menu_options_choice_inner_subtext">Ctrl+N</p>
                </div>
            </li>
            <li class="top_menu_options_li flex_container">
                <div class="flex_container">
                    <p class="top_menu_options_choice_inner_p_main_text">Open</p>
                </div>
                <div class="flex_container">
                    <p class="top_menu_options_choice_inner_subtext">Ctrl+O</p>
                </div>
            </li>
        </ul>
    </div>

Then, we can implement some basic styling in CSS.

style.css

.undecorated_list {
    list-style-type: none;
    padding: 0;
    margin: 0;
}
#top_menu_options_ul {
    height: 96%;
    width: 94%;
    padding-left: 3%;
    padding-right: 3%;
    padding-top: 2%;
    padding-bottom: 2%;
    overflow-y: scroll;
    scrollbar-width: none;
}
#top_menu_options_ul p {
    margin: 0;
    cursor: default;
}
#top_menu_options_ul li {
    justify-content: space-between;
}
#top_menu_options_ul li p.top_menu_options_choice_inner_subtext:not(p.disregard_subtext_formatting) { /*We don't want to make the right arrow dimmed and italic*/
    color: var(--subtle-color);
    font-style: italic;
}
#top_menu_options_ul li:not(#top_menu_options_ul li:first-child) {
    margin-top: clamp(var(--spacing-between-menu-items-min),var(--spacing-between-menu-items-preferred),var(--spacing-between-menu-items-max));
}
#top_menu_options_ul li:not(.not_a_context_menu_option):hover { /*This may be useful for labels and submenu headings within the context menu*/
    font-weight: bold;
}
#top_menu_options_ul li:not(.not_a_context_menu_option):hover p.top_menu_options_choice_inner_p_main_text {
    color: color-mix(in srgb, rgb(var(--accent-1)) 90%, var(--text-color) 10%);
}
.context_menu_header { /*Again, useful for submenu headings*/
    border-bottom: 2px solid var(--subtle-color);
    padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}

You may notice that we have three variables referenced here, namely --spacing-between-menu-items-preferred, --spacing-between-menu-items-min, and --spacing-between-menu-items-max. Before this CSS code, we first need to define these variables.

style.css

.dark {
  --subtle-color: rgb(133, 133, 133); /*This should change with theme, and that is why it was referenced above*/
}
.compact_spacing {
    --spacing-between-menu-items-min: 0px;
    --spacing-between-menu-items-preferred: 0px;
    --spacing-between-menu-items-max: 0px;
}
.comfy_spacing {
    --spacing-between-menu-items-min: 0px;
    --spacing-between-menu-items-preferred: 10px;
    --spacing-between-menu-items-max: 100vh;
}

In the future, we can change the class to increase or decrease the spacing between menu items.

Making the Buttons Do Stuff

To change the buttons for every menu item you click and make them do things when you click it, I made a separate file for this called top_menu_options.js.

In this file, I defined the log function and a placeholder function for things that will be implemented later. Then, added a reference to the inner unordered list of the top menu options in the DOM.

top_menu_options.js

function log(txt) {
    console.log('[top_menu.js][top_menu_options.js] ', txt); // 
}
log("Script loaded!");

const no_op = ()=>{}
const top_menu_options_ul = document.getElementById("top_menu_options_ul")

Then I came up with a structure for declaring the context menu items as an object.

Each key in this object corresponds to the text of the menu item that should be clicked for these specific items to show up. Every value is array of objects, with the objects this time representing each item in the context menu. There can be many different kinds of things you would want to put in here, so make sure to mark each of them with a type. I am using numbers to represent the types, and I am putting all the possible options in a JSDoc comment.

top_menu_options.js

/**
 * An object of menu options. For every option, there is another array of options (represented by objects). Below are the parameters each one can take.
 * - `type` (integer): Represents the type of the item
 *     - `0`: Represents a menu label with text. It is used to, for example, display the submenu breadcrumbs. It is only recommended to be used as the first menu item. 
 *     - `1`: Same as `0`, but with no text. This makes it just a separator.
 *     - `2`: A regular menu option that can be clicked to preform an action.
 *     - `3`: A submenu that can be clicked to open up more options.
 *     - `4`: Back header (used for properly displaying context menu breadcrumbs)
 *     - `5`: A checkbox item that can be toggled to on or off. The function given to `handler` is called every time it is clicked with a single argument passed to it - whether if the item has been changed to checked or not
 *  - `content` (string): The **unsanitized** HTML to display (not used on type `1`)
 *     - Note: on type `4` items, all the previous breadcrumbs will be dimmed and this will be displayed on the right side of the menu options
 *  - `subtext` (string): The **unsanitized** HTML to display on the right side in a darker color, typically a keyboard shortcut (only used on `2`)
 *  - `openTo` (array[object]): Another array of menu item objects that will be opened up when the current option is clicked (only used on `3`)
 *  - `handler` (function): A function to be run when the menu option is clicked (only used on `2` and `5`)
 *  - `checked` (boolean): Whether the menu item is checked by default or not (only used on `5`)
 *     - Note: This value changes. It is used by FeatherText to keep track of the current item's state
 *  - `disabled` (boolean): Whether the current option is disabled and cannot be used in the current scenario (only used on `2` and `5`)
 *     - Note: This is automatically enabled on type `3` items if all the type `2` items in `openTo` have this enabled. This cannot be set manually on type `3` items.
 *     - Note: If this is enabled on a type `5` item, it will remain locked in the current state whether it is checked or not
 */

Then right after, I begin defining the menu items.

top_menu_options.js

const menu_options = {
    "File":[],
    "Edit":[],
    "Go To":[],
    "Cursor":[],
    "View":[],
    "Options":[],
    "Help":[]
}

Let’s think ahead for a moment. If we want to open up a submenu, we will need to call the function to redraw the menu with a custom array and keep track of the previous items that the menu was opened to. Let’s define those two things.

top_menu_options.js

let previous_open_to = {}
function reset_previous_open_to() {previous_open_to={}}

function redraw_menu_from_array(arr) {}

Within our redraw_menu_from_array function, let’s first start out by clearing out the items that are currently in our list.

top_menu_options.js

top_menu_options_ul.innerHTML = ''

Then, we can loop over each item in the array.

top_menu_options.js

for (const [idx, element] of arr.entries()) {}

Within this for loop, we can chain if statements together to construct different types of menu items differently.

top_menu_options.js

if (element['type'] === 0) {}
else if (element['type'] === 1) {}
else if (element['type'] === 2) {}
else if (element['type'] === 3) {}
else if (element['type'] === 4) {}
else if (element['type'] === 5) {}

Constructing Type Zero Items

Type zero items are context menu headers. All the code in this section will be within the first if clause.

Let’s make a list item to contain all the text and then disable context menu option styling.

top_menu_options.js

const new_list_item = document.createElement("li")
new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option")

We don’t want top and bottom borders being applied if it is either the first or last elements, so let’s apply a different class for each scenario.

Scenario 1: The context menu header is the first item

top_menu_options.js

if (idx===0) {
  new_list_item.classList.add("context_menu_header")
}

Scenario 2: The context menu header is the last item

top_menu_options.js

else if (idx===items.length-1) { // idx is 0-indexed, while Array.prototype.length is not. This is why we subtract 1
  new_list_item.classList.add("context_menu_ending_header")
}

Scenario 3: Neither of the conditions above are true, meaning that the header is in the middle of the context menu.

top_menu_options.js

else {
  new_list_item.classList.add("context_menu_midway_header")
}

Let’s combine this into an if-else chain, and then add our list item to the outer unordered list.

top_menu_options.js

const new_list_item = document.createElement("li")
new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option")
if (idx===0) {
    new_list_item.classList.add("context_menu_header")
} else if (idx===items.length-1) {
    new_list_item.classList.add("context_menu_ending_header")
} else {
    new_list_item.classList.add("context_menu_midway_header")
}
top_menu_options_ul.appendChild(new_list_item)

Then, I made an element for the text and added it to the list item we created earlier.

top_menu_options.js

const new_list_header_text = document.createElement("p")
new_list_header_text.classList.add("top_menu_options_choice_inner_p_main_text")
new_list_header_text.innerHTML = element['content']
new_list_item.appendChild(new_list_header_text)

Now, we can update our CSS rules for the context menu headers.

style.css

.context_menu_header {
    border-bottom: 2px solid var(--subtle-color);
    padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
.context_menu_midway_header {
    border-bottom: 2px solid var(--subtle-color);
    border-top: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    margin-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
.context_menu_ending_header {
    border-top: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}

I also updated it to give it even spacing.

Constructing Type One Items

Type one items are simply context menu separators, so this one is fairly straightforward. All code in this section will be contained inside our second if clause.

Let’s start by making a new list item.

top_menu_options.js

const new_list_item = document.createElement("li")

It isn’t an actual clickable menu item, so let’s add some identifiers to reflect these properties in the CSS we will add later.

top_menu_options.js

new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option", "context_menu_separator")

Then, we can add this list item to our unordered list.

top_menu_options.js

top_menu_options_ul.appendChild(new_list_item)

Finally, we can give even spacing to our context menu separator with a new CSS rule.

style.css

.context_menu_separator {
    border-bottom: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    margin-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}

Constructing Type Two Items

Type two items are probably the most common type I will use because this represents a regular item that you can click on to perform an action. The code here will be part of the third clause.

Let’s start by making a new list item, adding the relevant classes to it, and then attaching it to the handler.

top_menu_options.js

const new_list_item = document.createElement("li")
new_list_item.classList.add("top_menu_options_li", "flex_container")
if (!element['disabled']) {new_list_item.addEventListener("click", element['handler'])}
top_menu_options_ul.appendChild(new_list_item)

Then, we can make the divs that go on both the left and right sides of the list item.

top_menu_options.js

const left_side_div = document.createElement("div")
left_side_div.classList.add("flex_container")
new_list_item.appendChild(left_side_div)
const right_side_div = document.createElement("div")
right_side_div.classList.add("flex_container")
new_list_item.appendChild(right_side_div)

We can then make the text that goes in either div.

top_menu_options.js

const left_side_text = document.createElement("p")
left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
left_side_text.innerHTML = element['content']
left_side_div.appendChild(left_side_text)
const right_side_text = document.createElement("p")
right_side_text.innerHTML = element['subtext']
right_side_text.classList.add("top_menu_options_choice_inner_subtext")
right_side_div.appendChild(right_side_text)

We then need to provide visual indication of the item being disabled, so let’s add a class to the list item.

top_menu_options.js

if (element['disabled']) {
    new_list_item.classList.add("context_menu_option_disabled")
}

We can then add the ruleset for context_menu_option_disabled.

style.css

.context_menu_option_disabled {
    color: var(--subtle-color);
}
.context_menu_option_disabled p.top_menu_options_choice_inner_subtext {
    color: color-mix(in srgb, var(--subtle-color) 75%, var(--window-bg-color) 25%) !important;
}]

Constructing Type Three Items

Type three items are the buttons that lead to a submenu. Rather than draw a new box, I think it would be better to do what Firefox does and show the new menu options without spawning a new box. All code will be in the fourth if clause.

We can structure this the exact same as type two items, but they do a different thing when they are clicked. We also want to stop the regular subtext formatting from being applied to this element.

We are creating a new list item below.

top_menu_options.js

const new_list_item = document.createElement("li")
new_list_item.classList.add("top_menu_options_li", "flex_container")
top_menu_options_ul.appendChild(new_list_item)

Let’s create the left side div and p elements.

top_menu_options.js

const left_side_div = document.createElement("div")
left_side_div.classList.add("flex_container")
new_list_item.appendChild(left_side_div)

const left_side_text = document.createElement("p")
left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
left_side_text.innerHTML = element['content']
left_side_div.appendChild(left_side_text)

Let’s create the right-side div. Again, this will be the exact same as a type two item.

top_menu_options.js

const right_side_div = document.createElement("div")
right_side_div.classList.add("flex_container")
new_list_item.appendChild(right_side_div)

Then for the right-side text, we will do something a little different. We will use a right quote (›) character instead of subtext defined in the menu_options object. We will also prevent the regular subtext formatting from being applied by adding another class to this text, disregard_subtext_formatting.

top_menu_options.js

const right_side_text = document.createElement("p")
right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting")
right_side_text.innerHTML = '›'
right_side_div.appendChild(right_side_text)

Then, we can handle a click event by stopping all other events (to make sure that the menu item does not lose focus and disappear), and setting the current array as the one previously opened because in the next line, we redraw the context menu items with the elements that are supposed to be within the submenu.

top_menu_options.js

new_list_item.addEventListener("click",e=>{
    e.stopImmediatePropagation()
    previous_open_to = arr
    redraw_menu_from_array(element['openTo'])
})

Constructing Type Four Items

Type four items are the headers that allow you to open a submenu’s parent. All code in this section will be in the fifth clause.

Because we need to make sure this option is as visible as possible to the user in case they clicked on the submenu by mistake, it must be the first item in the list of context menu items. We can run a quick check to see if this is true, after which we will log an error message and skip drawing this item if it is not.

top_menu_options.js

if (idx !== 0) {log("Type 4 elements must be the first item of the array!");continue;}

Then, we can make a list item and have it show the previously opened menu on click.

top_menu_options.js

new_list_item.addEventListener("click",e=>{
    e.stopImmediatePropagation()
    redraw_menu_from_array(previous_open_to)
    previous_open_to = arr
})

We can also give it the typical classes of a list item.

top_menu_options.js

new_list_item.classList.add("top_menu_options_li", "flex_container")
top_menu_options_ul.appendChild(new_list_item)

We construct the left and right side divs like we normally do, except the left side div is a label for the entire list item being a back button.

top_menu_options.js

const left_side_div = document.createElement("div")
left_side_div.classList.add("flex_container")
new_list_item.appendChild(left_side_div)

const left_side_text = document.createElement("p")
left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
left_side_text.innerHTML = '‹ Back'
left_side_div.appendChild(left_side_text)

const right_side_div = document.createElement("div")
right_side_div.classList.add("flex_container")
new_list_item.appendChild(right_side_div)

For the right-side text, we are going to do something interesting. We are going to make the element, as usual, and then dim its content, except for the final slash. We will do this by using a regular expression to wrap everything that needs to be dimmed in a span tag.

top_menu_options.js

const right_side_text = document.createElement("p")
right_side_text.innerHTML = element['content'].replace(/^(.*\/)/, '<span style="color: var(--subtle-color);">$1</span>');
right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting") // We want to remove as much default CSS styling as possible, which is why we do not apply the normal formatting for subtext
right_side_div.appendChild(right_side_text)

Constructing Type Five Items

Type five items are the checkbox items that turn something on or off. All code in this section will be in the sixth and final if clause.

We can define the new list item along with the left and right side divs and texts like we would normally do.

top_menu_options.js

const new_list_item = document.createElement("li")
new_list_item.classList.add("top_menu_options_li", "flex_container")
top_menu_options_ul.appendChild(new_list_item)

const left_side_div = document.createElement("div")
left_side_div.classList.add("flex_container")
new_list_item.appendChild(left_side_div)

const left_side_text = document.createElement("p")
left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
left_side_text.innerHTML = element['content']
left_side_div.appendChild(left_side_text)

const right_side_div = document.createElement("div")
right_side_div.classList.add("flex_container")
new_list_item.appendChild(right_side_div)

const right_side_text = document.createElement("p")
right_side_text.innerHTML = element['subtext']
right_side_text.classList.add("top_menu_options_choice_inner_subtext")
right_side_div.appendChild(right_side_text)

Before the text on the left side, we should put a box to indicate whether or not the current option is selected. This will be a div element that will be filled if the option is selected, and only show the border if the option is not selected.

top_menu_options.js

const left_side_selection_indicator = document.createElement("div")
left_side_selection_indicator.classList.add("context_menu_option_selection_indicator")
if (element['checked']) {left_side_selection_indicator.classList.add("context_menu_option_selection_indicator_selected")}
left_side_text.insertAdjacentElement("beforebegin",left_side_selection_indicator)

We also have to give indication of the item being disabled if it is, so let’s implement that logic here.

top_menu_options.js

if (element['disabled']) {
    new_list_item.classList.add("context_menu_option_disabled")
}

If the menu option isn’t disabled, then we can toggle its state by changing its context menu indicator.

top_menu_options.js

new_list_item.addEventListener("click", ()=>{
  if (!element['disabled']) {
      element['checked'] = !element['checked']
      left_side_selection_indicator.classList.toggle("context_menu_option_selection_indicator_selected")
      element['handler'](element['checked'])
  }
})

Then, using CSS, we can make the context menu indicator however we want. In this case, it will be a small div that can be filled to indicate that the menu option is selected, or empty to indicate that the menu option is not selected.

style.css

.context_menu_option_selection_indicator {
    height: 9px;
    aspect-ratio: 1; /* Makes the height and width equal */
    align-self: center; /* Centers itself vertically */
    border: 2px solid var(--text-color);
    margin-right: 8px; /* Gives space between the indicator and the text */
}
.context_menu_option_selection_indicator_selected {
    background-color: var(--text-color);
}
.context_menu_option_disabled .context_menu_option_selection_indicator {
    border-color: var(--subtle-color);
}
.context_menu_option_disabled .context_menu_option_selection_indicator_selected {
    background-color: var(--subtle-color) !important;
}

Redrawing The Menu Items Based on a Menu Option

In order to do this, we can draw the menu options for that specified menu item.

top_menu_options.js

export function redraw_menu_options(menu_item) {
    if (menu_options[menu_item] === undefined) {log("Not a valid menu option!");return;}
    
    redraw_menu_from_array(menu_options[menu_item])
}

Structuring The Program

Notice how we added export to the beginning of the function declaration. This is because of the way I am going to structure this program. We will import the function from this file and use it in top_menu_bar.js. This code will be at the beginning of top_menu_bar.js. I will be moving both of those files into a separate subfolder called top_menu.

top_menu_bar.js

import { redraw_menu_options, reset_previous_open_to } from "./top_menu_options.js";

In top_menu_options.js, reset_previous_open_to is not exported from the file. Let’s change that by adding the export keyword to the function declaration at the beginning of the file.

top_menu_options.js

export function reset_previous_open_to() {previous_open_to={}}

Going back to top_menu_bar.js, we can now put the following line as the final line before we close the click and mouseenter listeners in our menu item loop.

top_menu_bar.js

redraw_menu_options(top_menu_item.innerText)

Then at the click for the entire document, we can put this line right before we close the if statement.

top_menu_bar.js

reset_previous_open_to()

Then, I renamed top_menu_bar.js to top_menu.js, so that our directory tree looks like this.

Plaintext

C:.
│   index.html
│   style.css
│   
└───scripts
    └───top_menu
            top_menu.js
            top_menu_options.js

We should represent these changes in index.html by replacing the script import within the head tags with the code below.

index.html

<script src="scripts/top_menu/top_menu.js" type="module" defer></script>

Final Code

Assuming everything in the guide was followed (along with some refactoring and style changes), this is how your code should look.

index.html

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="style.css">
    <title>FeatherText</title>
    <script src="scripts/top_menu/top_menu.js" type="module" defer></script>
</head>

<body class="dark">
    <div id="top_menu_container" class="fullwidth flex_container">
        <div class="top_menu_item">
            <p class="top_menu_item_text">File</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">Edit</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">Go To</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">Cursor</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">View</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">Options</p>
        </div>
        <div class="top_menu_item">
            <p class="top_menu_item_text">Help</p>
        </div>
    </div>
    <div id="top_menu_options" class="comfy_spacing">
        <ul id="top_menu_options_ul" class="undecorated_list">
            
                
        </ul>
    </div>
    <div>
    </div>
</body>

</html>

style.css

/* Import "Azeret Mono" font (to be used on UI elements like the menu bar) */
@import url('https://fonts.googleapis.com/css2?family=Azeret+Mono:ital,wght@0,100..900;1,100..900&display=swap');

/* Import "DM Mono" font (to be used in the text box) */
@import url('https://fonts.googleapis.com/css2?family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&display=swap');

.dark {
    --window-bg-color: #010022;
    --context-bg-color: 46, 46, 46  ; /*#313131*/
    --text-color: white;
    --subtle-color: rgb(173, 173, 173);
    --accent-1: 0, 136, 255; /*#0088ff*/
}
.compact_spacing {
    --spacing-between-menu-items-min: 0px;
    --spacing-between-menu-items-preferred: 0px;
    --spacing-between-menu-items-max: 0px;
}
.comfy_spacing {
    --spacing-between-menu-items-min: 0px;
    --spacing-between-menu-items-preferred: 10px;
    --spacing-between-menu-items-max: 100vh;
}
body {
    background-color: var(--window-bg-color);
    color: var(--text-color);
    height: 100vh;
    width: 100vw;
    margin: 0;
    overflow: hidden;
    font-family: 'Azeret Mono';
}

.fullwidth {width: 100vw;}
.flex_container {display: flex;}

/* MARK: Top Menu Bar
*/
#top_menu_container {
    padding: 2px;
    margin-bottom: 10px;
    flex-wrap: wrap;
}
#top_menu_container *:first-child {
    padding-left: 10px;
}
#top_menu_container *:not(#top_menu_container *:first-child) {
    padding-left: clamp(4px, 0.75vw, 100vw);
}
#top_menu_container * {
    margin: 0;
    margin-top: 10px;
}
.top_menu_item {
    cursor: default;
}
#top_menu_container p.top_menu_item_text {
    clip-path: polygon(
        0 0,
        100% 0,
        100% calc(100% - 4px),
        0 calc(100% - 4px)
    );
    transition: clip-path 50ms linear;
    transition-delay: 50ms;
    margin: 0;
    padding: 0;
    border-bottom: 6px solid var(--text-color);
}
#top_menu_container .top_menu_item:hover p.top_menu_item_text, #top_menu_container p.force_selected_border_underline, #top_menu_container p.force_white_border_underline {
    clip-path: polygon(
        0 0,
        100% 0,
        100% 100%,
        0 100%
    );
}
#top_menu_container p.force_selected_border_underline {border-bottom-color: rgb(var(--accent-1)) !important;}
#top_menu_container p.force_white_border_underline {border-bottom-color: var(--text-color);}
/* MARK: Top Menu Options
*/
#top_menu_options {
    width: clamp(200px, 20vw, 300px);
    height: clamp(200px, 30vh, 500px);
    position: absolute;
    /* display: none; */
    transform: translateY(-3%);
    z-index: 2;
    position: absolute;
    backdrop-filter: blur(5px);
    background-color: rgba(var(--context-bg-color),0.7);
    overflow-y: scroll;
    scrollbar-width: none;
}
.undecorated_list {
    list-style-type: none;
    padding: 0;
    margin: 0;
}
#top_menu_options_ul {
    height: 96%;
    width: 94%;
    padding-left: 3%;
    padding-right: 3%;
    padding-top: 2%;
    padding-bottom: 2%;
    overflow-y: scroll;
    scrollbar-width: none;
}
#top_menu_options_ul p {
    margin: 0;
    cursor: default;
}
#top_menu_options_ul li {
    justify-content: space-between;
}
#top_menu_options_ul li p.top_menu_options_choice_inner_subtext:not(p.disregard_subtext_formatting) {
    color: var(--subtle-color);
    font-style: italic;
}
#top_menu_options_ul li:not(#top_menu_options_ul li:first-child) {
    margin-top: clamp(var(--spacing-between-menu-items-min),var(--spacing-between-menu-items-preferred),var(--spacing-between-menu-items-max));
}
#top_menu_options_ul li:not(.not_a_context_menu_option):not(.context_menu_option_disabled):hover {
    font-weight: bold;
}
#top_menu_options_ul li:not(.not_a_context_menu_option):not(.context_menu_option_disabled):hover p.top_menu_options_choice_inner_p_main_text {
    color: color-mix(in srgb, rgb(var(--accent-1)) 90%, var(--text-color) 10%);
}
.context_menu_option_disabled {
    color: var(--subtle-color);
}
.context_menu_option_disabled p.top_menu_options_choice_inner_subtext {
    color: color-mix(in srgb, var(--subtle-color) 75%, var(--window-bg-color) 25%) !important;
}
.context_menu_header {
    border-bottom: 2px solid var(--subtle-color);
    padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
.context_menu_midway_header {
    border-bottom: 2px solid var(--subtle-color);
    border-top: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    margin-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
.context_menu_ending_header {
    border-top: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    padding-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
.context_menu_separator {
    border-bottom: 2px solid var(--subtle-color);
    margin-top: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
    margin-bottom: clamp(var(--spacing-between-menu-items-min), var(--spacing-between-menu-items-preferred), var(--spacing-between-menu-items-max));
}
#top_menu_options_ul li:not(#top_menu_options_ul li:has(div.context_menu_option_selection_indicator)) {
    margin-left: 20px;
}
.context_menu_option_selection_indicator {
    height: 9px;
    aspect-ratio: 1;
    align-self: center;
    border: 2px solid var(--text-color);
    margin-right: 8px;
}
.context_menu_option_selection_indicator_selected {
    background-color: var(--text-color);
}
.context_menu_option_disabled .context_menu_option_selection_indicator {
    border-color: var(--subtle-color);
}
.context_menu_option_disabled .context_menu_option_selection_indicator_selected {
    background-color: var(--subtle-color) !important;
}

top_menu.js

function log(txt) {
    console.log('[top_menu.js] ', txt);
}
log("Script loaded!");

import { redraw_menu_options, reset_previous_open_to } from "./top_menu_options.js";

const context_menu_state = { clicked: "", menu_options_clicked: false };
const menu_bar_items = {};
const menu_bar_inner_p_items = {};
const context_menu_container = document.getElementById("top_menu_options");

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {
    menu_bar_items[top_menu_item.innerText] = top_menu_item;
    menu_bar_inner_p_items[top_menu_item.getElementsByClassName("top_menu_item_text")[0].innerText] = top_menu_item.getElementsByClassName("top_menu_item_text")[0];
});

Array.from(document.getElementsByClassName("top_menu_item")).forEach(top_menu_item => {
    const top_menu_item_inner_p = top_menu_item.getElementsByClassName("top_menu_item_text")[0];

    top_menu_item.addEventListener("click", () => {
        context_menu_container.style.display = "block";
        context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";
        context_menu_state.clicked = top_menu_item.innerText;
        Object.values(menu_bar_inner_p_items).forEach(val => {
            val.classList.remove("force_selected_border_underline");
        });
        top_menu_item_inner_p.classList.add("force_selected_border_underline");
        redraw_menu_options(top_menu_item.innerText)
    });

    top_menu_item.addEventListener("mouseenter", () => {
        if (context_menu_state.clicked !== "") {
            context_menu_container.style.display = "block";
            context_menu_container.style.left = top_menu_item_inner_p.offsetLeft + "px";
            Object.values(menu_bar_inner_p_items).forEach(val => {
                val.classList.remove("force_white_border_underline");
            });
            if (context_menu_state.clicked !== top_menu_item.innerText) {
                menu_bar_inner_p_items[top_menu_item.innerText].classList.add("force_white_border_underline");
            }
        }
        redraw_menu_options(top_menu_item.innerText)
    });
});

document.addEventListener("click", (e) => {
    if (!context_menu_container.contains(e.target) && !Object.values(menu_bar_items).some(item => item.contains(e.target))) {
        context_menu_state.clicked = "";
        context_menu_container.style.display = "none";
        Object.values(menu_bar_inner_p_items).forEach(menu_bar_inner_p => {
            menu_bar_inner_p.classList.remove("force_selected_border_underline", "force_white_border_underline");
        });
        reset_previous_open_to()
    }
});

top_menu_options.js

function log(txt) {
    console.log('[top_menu.js][top_menu_options.js] ', txt);
}
log("Script loaded!");

const no_op = ()=>{}
const top_menu_options_ul = document.getElementById("top_menu_options_ul")

/**
 * An object of menu options. For every option, there is another array of options (represented by objects). Below are the parameters each one can take.
 * - `type` (integer): Represents the type of the item
 *     - `0`: Represents a menu label with text. It is used to, for example, display the submenu breadcrumbs. It is only recommended to be used as the first menu item. 
 *     - `1`: Same as `0`, but with no text. This makes it just a separator.
 *     - `2`: A regular menu option that can be clicked to preform an action.
 *     - `3`: A submenu that can be clicked to open up more options.
 *     - `4`: Back header (used for properly displaying context menu breadcrumbs)
 *     - `5`: A checkbox item that can be toggled to on or off. The function given to `handler` is called every time it is clicked with a single argument passed to it - whether if the item has been changed to checked or not
 *  - `content` (string): The **unsanitized** HTML to display (not used on type `1`)
 *     - Note: on type `4` items, all the previous breadcrumbs will be dimmed and this will be displayed on the right side of the menu options
 *  - `subtext` (string): The **unsanitized** HTML to display on the right side in a darker color, typically a keyboard shortcut (only used on `2`)
 *  - `openTo` (array[object]): Another array of menu item objects that will be opened up when the current option is clicked (only used on `3`)
 *  - `handler` (function): A function to be run when the menu option is clicked (only used on `2` and `5`)
 *  - `checked` (boolean): Whether the menu item is checked by default or not (only used on `5`)
 *     - Note: This value changes. It is used by FeatherText to keep track of the current item's state
 *  - `disabled` (boolean): Whether the current option is disabled and cannot be used in the current scenario (only used on `2` and `5`)
 *     - Note: This is automatically enabled on type `3` items if all the type `2` items in `openTo` have this enabled. This cannot be set manually on type `3` items.
 *     - Note: If this is enabled on a type `5` item, it will remain locked in the current state whether it is checked or not. It can be changed programatically, but not by the user
 */
const menu_options = {
    "File":[],
    "Edit":[],
    "Go To":[],
    "Cursor":[],
    "View":[],
    "Options":[],
    "Help":[]
}

let previous_open_to = {}
export function reset_previous_open_to() {previous_open_to={}}

function redraw_menu_from_array(arr) {
    // Clear existing items
    top_menu_options_ul.innerHTML = ''
    
    // Add new items
    for (const [idx, element] of arr.entries()) {
        if (element['type'] === 0) {
            const new_list_item = document.createElement("li")
            new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option")
            if (idx===0) {
                new_list_item.classList.add("context_menu_header")
            } else if (idx===items.length-1) {
                new_list_item.classList.add("context_menu_ending_header")
            } else {
                new_list_item.classList.add("context_menu_midway_header")
            }
            top_menu_options_ul.appendChild(new_list_item)
            
            const new_list_header_text = document.createElement("p")
            new_list_header_text.classList.add("top_menu_options_choice_inner_p_main_text")
            new_list_header_text.innerHTML = element['content']
            new_list_item.appendChild(new_list_header_text)
        } else if (element['type'] === 1) {
            const new_list_item = document.createElement("li")
            new_list_item.classList.add("top_menu_options_li", "not_a_context_menu_option", "context_menu_separator")
            top_menu_options_ul.appendChild(new_list_item)
        } else if (element['type'] === 2) {
            const new_list_item = document.createElement("li")
            new_list_item.classList.add("top_menu_options_li", "flex_container")
            if (!element['disabled']) {new_list_item.addEventListener("click", element['handler'])}
            top_menu_options_ul.appendChild(new_list_item)
            
            const left_side_div = document.createElement("div")
            left_side_div.classList.add("flex_container")
            new_list_item.appendChild(left_side_div)
            
            const left_side_text = document.createElement("p")
            left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
            left_side_text.innerHTML = element['content']
            left_side_div.appendChild(left_side_text)
            
            const right_side_div = document.createElement("div")
            right_side_div.classList.add("flex_container")
            new_list_item.appendChild(right_side_div)
            
            const right_side_text = document.createElement("p")
            right_side_text.innerHTML = element['subtext']
            right_side_text.classList.add("top_menu_options_choice_inner_subtext")
            right_side_div.appendChild(right_side_text)
            
            if (element['disabled']) {
                new_list_item.classList.add("context_menu_option_disabled")
            }
        } else if (element['type'] === 3) {
            const new_list_item = document.createElement("li")
            new_list_item.classList.add("top_menu_options_li", "flex_container")
            top_menu_options_ul.appendChild(new_list_item)
            
            const left_side_div = document.createElement("div")
            left_side_div.classList.add("flex_container")
            new_list_item.appendChild(left_side_div)
            
            const left_side_text = document.createElement("p")
            left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
            left_side_text.innerHTML = element['content']
            left_side_div.appendChild(left_side_text)
            
            const right_side_div = document.createElement("div")
            right_side_div.classList.add("flex_container")
            new_list_item.appendChild(right_side_div)
            
            const right_side_text = document.createElement("p")
            right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting")
            right_side_text.innerHTML = '›'
            right_side_div.appendChild(right_side_text)
            
            new_list_item.addEventListener("click",e=>{
                e.stopImmediatePropagation()
                previous_open_to = arr
                redraw_menu_from_array(element['openTo'])
            })
        } else if (element['type'] === 4) {
            if (idx !== 0) {log("Type 4 elements must be the first item of the array!");continue;}
            const new_list_item = document.createElement("li")
            new_list_item.addEventListener("click",e=>{
                e.stopImmediatePropagation()
                redraw_menu_from_array(previous_open_to)
                previous_open_to = arr
            })
            new_list_item.classList.add("top_menu_options_li", "flex_container")
            top_menu_options_ul.appendChild(new_list_item)
            
            const left_side_div = document.createElement("div")
            left_side_div.classList.add("flex_container")
            new_list_item.appendChild(left_side_div)
            
            const left_side_text = document.createElement("p")
            left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
            left_side_text.innerHTML = '‹ Back'
            left_side_div.appendChild(left_side_text)
            
            const right_side_div = document.createElement("div")
            right_side_div.classList.add("flex_container")
            new_list_item.appendChild(right_side_div)
            
            const right_side_text = document.createElement("p")
            right_side_text.innerHTML = element['content'].replace(/^(.*\/)/, '<span style="color: var(--subtle-color);">$1</span>');
            right_side_text.classList.add("top_menu_options_choice_inner_subtext", "disregard_subtext_formatting")
            right_side_div.appendChild(right_side_text)
        } else if (element['type'] === 5) {
            const new_list_item = document.createElement("li")
            new_list_item.classList.add("top_menu_options_li", "flex_container")
            top_menu_options_ul.appendChild(new_list_item)
            
            const left_side_div = document.createElement("div")
            left_side_div.classList.add("flex_container")
            new_list_item.appendChild(left_side_div)
            
            const left_side_text = document.createElement("p")
            left_side_text.classList.add("top_menu_options_choice_inner_p_main_text")
            left_side_text.innerHTML = element['content']
            left_side_div.appendChild(left_side_text)
            
            const left_side_selection_indicator = document.createElement("div")
            left_side_selection_indicator.classList.add("context_menu_option_selection_indicator")
            if (element['checked']) {left_side_selection_indicator.classList.add("context_menu_option_selection_indicator_selected")}
            left_side_text.insertAdjacentElement("beforebegin",left_side_selection_indicator)
            
            const right_side_div = document.createElement("div")
            right_side_div.classList.add("flex_container")
            new_list_item.appendChild(right_side_div)
            
            const right_side_text = document.createElement("p")
            right_side_text.innerHTML = element['subtext']
            right_side_text.classList.add("top_menu_options_choice_inner_subtext")
            right_side_div.appendChild(right_side_text)
            
            if (element['disabled']) {
                new_list_item.classList.add("context_menu_option_disabled");
            }
            new_list_item.addEventListener("click", ()=>{
                if (!element['disabled']) {
                    element['checked'] = !element['checked']
                    left_side_selection_indicator.classList.toggle("context_menu_option_selection_indicator_selected")
                    element['handler'](element['checked'])
                }
            })
        }
    }
}

export function redraw_menu_options(menu_item) {
    if (menu_options[menu_item] === undefined) {log("Not a valid menu option!");return;}
    
    redraw_menu_from_array(menu_options[menu_item])
}

redraw_menu_options("File")

Monday, March 3, 2025, 9:04 PM (GMT+0000)

Programming a Basic Text Editor in HTML, CSS, and JS
When you are looking to quickly edit a file, chances are you open something like Notepad or Notepad++. However, on more limiting devices such as a phone or a Chromebook with no built-in text editor (Google Docs is a word processor, so it doesn’t count), your best bet is searching something for an online text editor. The issue with this is that most online text editors are sloppy and don’t get the same features as you do on desktop (and the ones that do feel very overwhelming)

The Idea

Here, I will be refreshing my memory on the three web technologies by creating a text editor that is reminiscent of Vim and Emacs. It should have a tab bar like Emacs, the commands of Vim, and full keyboard support like both. Supporting Vim Script or the macros that gives Emacs its name is out of the scope of this guide, especially for an online text editor. However, they may be supported in a future update.

I will also be using pure HTML, CSS, and JS. No preprocessors or frameworks used here. This is because I am documenting my learning process here for those three technologies, not for external UI frameworks.

One important thing to note is that this text editor will only have two modes, “insert” and “view,” and the default mode will be insert. This is because while I do want to make sure my editor is powerful and familiar to Vim and Emacs users alike, it should also be user-friendly. In insert mode, you can edit the text directly, but if you press ESC and end up in viewing mode, you can use Vim-like commands and control the top menu bar with your keyboard, like you can on Emacs.

Starting Off

I started off with a single file, index.html, where I put in the standard boilerplate and then set the title of the webpage to what this text editor will be named: FeatherText.

Then I linked a CSS file called style.css.
index.html
```

<link rel="stylesheet" href="style.css">
```
Within that style.css file, I then started by using CSS variables to define a starting theme (did I mention that this text editor will have multiple themes?)
style.css
```
.dark {
    --window-bg-color: #010022;
    --text-color: white;
}
```
Then, I had to link the theme to the webpage by adding the dark class to the body element. This way, the dark theme will be enabled by default.
index.html
```

<body class="dark">
```
Afterwards, I removed the scrollbars (since they were just going to cause problems down the line) and applied the dark theme to the webpage by setting the background color and text color to match with the one defined in the theme.
style.css
```
body {
    background-color: var(--window-bg-color);
    color: var(--text-color);
    height: 100vh;
    width: 100vw;
    margin: 0;
    overflow: hidden;
}
```
Then, I added two fonts to be applied to the entire webpage in different UI elements.
style.css
```
/* PUT THESE LINES AT THE VERY TOP OF STYLE.CSS */

/* Import "Azeret Mono" font (to be used on UI elements like the menu bar) */
@import url('https://fonts.googleapis.com/css2?family=Azeret+Mono:ital,wght@0,100..900;1,100..900&display=swap');

/* Import "DM Mono" font (to be used in the text box) */
@import url('https://fonts.googleapis.com/css2?family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&display=swap');
```
Top Menu Bar

To implement a top menu bar, similar to the one you get in Emacs, I started off with a div full of other divs (this time our menu items), so our HTML file would look something like this:
index.html
```
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="style.css">
    <title>FeatherText</title>
</head>
<body class="dark">
    <div id="top_menu_container" class="fullwidth flex_container">
        <div class="top_menu_item">File</div>
        <div class="top_menu_item">Edit</div>
        <div class="top_menu_item">Go To</div>
        <div class="top_menu_item">Cursor</div>
        <div class="top_menu_item">View</div>
        <div class="top_menu_item">Options</div>
        <div class="top_menu_item">Help</div>
    </div>
</body>
</html>
```
Let’s give this top menu bar some padding:
style.css
```
#top_menu_container {
    padding: 2px;
    flex-wrap: wrap;
}
```
Now, let’s give each item (except for the first) a minimum of 23 pixels of left margin, but preferably 1.5% of the total width of the screen. Let’s give our first item 10 pixels of left margin.
CSS
```
#top_menu_container *:not(#top_menu_container *:first-child) {
    margin-left: clamp(23px, 1.5vw, 100vw);
}
#top_menu_container:first-child {
    margin-left: 10px;
}
```
Let’s give every item a top margin of 10 pixels and clear out any other default margins set by the browser, while making sure that our cursor remains the default (this is just in case we decide to add a link to the top menu bar, which would change the cursor to a pointer).
CSS
```
#top_menu_container * {
    margin: 0;
    margin-top: 10px;
}
.top_menu_item {
    cursor: default;
}
```
Then, let’s add a border at the bottom on hover for every top menu item to give indication of what is being hovered on.
CSS
```
#top_menu_container div.top_menu_item {
    transition: border-bottom linear 50ms;
}
#top_menu_container div.top_menu_item:hover {
    border-bottom: 4px solid var(--text-color);
}
```
But wait! You may notice a problem with what I have done. While nothing seems wrong, try putting some sample text below the menu bar and see what happens below.

You probably noticed that the bottom border transition shifted the text downwards, which I would say is pretty distracting and does not look good at all. This is because the border-bottom property affects the size of the element. There are some ways to get the same effect without affecting layout, but the simplest one to explain (and the one I am using here) is to keep the border on all the time but only show it on hover.

The clip-path property essentially allows you to crop an element to a specific viewable portion. Better yet – you can animate it.

What we will be doing is keeping the border always on but set the clip-path property such that the border is hidden. Then, we can animate the clip-path property to show the full shape on hover. Let’s replace our CSS rulesets to use this approach.
style.css
```
#top_menu_container div.top_menu_item {
clip-path: polygon(
        /* Top left corner */ 0 0,
        /* Top right corner */ 100% 0,
        /* Bottom right corner */ 100% calc(100% - 4px),
        /* Bottom left corner */ 0 calc(100% - 4px) /* Element full width, minus the border width*/
  );
transition: clip-path 50ms linear; /* Animate it */
 border-bottom: 4px solid var(--text-color);
 }
```
You will notice that the first number of every pair passed into the polygon function is 0, except for the top and bottom right corners, which are 100%. This represents the x-index for our polygon, but because we are not cropping anything from the sides, this can represent the full width.

And then on hover, we show the full shape to reveal the border.
CSS
```
#top_menu_container div.top_menu_item:hover {
    clip-path: polygon(
        0 0,
        100% 0,
        100% 100%,
        0 100%
    );
}
```
Coding the Menu Options

These menu buttons are cool, but they don’t do anything yet. So, I created a folder called scripts and a file in that folder called top_menu_bar_options.js.

Right now, we have the basic look and feel of a traditional menu bar app, but to make it functional, we will need to use some JavaScript. Stay tuned for a part 2!
Thursday, January 30, 2025, 10:38 PM (GMT+0000)

How Does Base64 Encoding Work?

Some things you might have heard being thrown around in the cryptography space are encoding methods. Such encoding methods are normally used to turn binary data into text, typically so it can be used to transmit data over communication methods that do not allow raw binary data to be transmitted.

How does binary turn into Base64?

First, let’s say that we want to encode an image. We start by reading all the bits in the image, which, for this example, will result in a binary stream of 010010101101001000111111. It is important to note that most images are a lot larger than this and will contain hundreds of thousands of bits.

The encoder will first break the binary stream up into their own chunks, each containing six bits. We can do this to our example and get 010010 101101 001000 111111.

Then, we turn our binary chunks into decimals.

To do this, working right to left, we multiply each bit by 2 to the power of p, where p starts from zero and increments by one for every bit in the chunk.

For example, we can calculate 010010 by performing the expression (0 * 2⁰) + (1 * 2¹) + (0 * 2²) + (0 * 2³) + (1 * 2⁴) + (0 * 2⁵). We can strip away any parts that multiply by zero, since those will always be zero. The simplified equation is (1 * 2¹) + (1 * 2⁴) = 2 + 16 = 18. Therefore, 010010 is equivalent to the decimal 18.

Turning every single six-bit binary chunk into a decimal using the process shown above constitutes the numbers (in left-to-right order) 18, 45, 8, and 63.

After that, we can simply cross reference the numbers with whatever alphabet we should use. For example, the six-bit chunk 010010 turns into the decimal 18, which corresponds to S on the Base64 standard alphabet. Therefore, 010010 101101 001000 111111 turns into StI/.

How does binary turn into Base64?: fixing the padding

If the binary string is a multiple of 3, everything fits perfectly. However, if it is not, then we need to add padding to the encoded string until the string’s bit count becomes a multiple of 3.

Let’s take 010010 101101 001000 111111 01 as an example. It is 26 bits long and all full six-bit chunks encode to StI/. However, this is leaving out the two bits at the end.

To pad this, we will keep adding zeroes to the end of the string until it can become a full chunk. Thus, the final chunk, 01, becomes 010000, which corresponds to Q in the Base64 alphabet. Then, we add one padding character (typically an =) for every two bits we had to add for the string to only be made of full chunks. We add two padding characters because four (the number of bits we had to add) divided by two is two.

Thus, you would think that the padded string would be StI/Q==, right? Well, no, because that extra character adds another six bits, meaning that there are still two bits at the end. We can only stay in such a loop two times, so we need to add more zeroes. That means that the final letter is 000000, which is A.

So, the final result is StI/QA==.

How does Base64 turn into binary?

Base64 decoding is just like base64 encoding, but in reverse. First, we turn every single non-padding character into its respective index on the alphabet, then we turn every single number into its binary equivalent.

For example, let’s take our string StI/QA==. We first take every single non-padding character and turn it into a string of numbers that represents their indexes on the alphabet.

This turns our Base64-encoded string into a string of numbers, specifically 18 45 37 62 16 0 = =, since we don’t turn the padding into a number

The Standard Base64 Alphabet

0: A	7: H	14: O	21: V	28: c	35: j	42: q	49: x	56: 5
1: B	8: I	15: P	22: W	29: d	36: k	43: r	50: y	57: 6
2: C	9: J	16: Q	23: X	30: e	37: l	44: s	51: z	58: 7
3: D	10: K	17: R	24: Y	31: f	38: m	45: t	52: 1	59: 8
4: E	11: L	18: S	25: Z	32: g	39: n	46: u	53: 2	60: 9
5: F	12: M	19: T	26: a	33: h	40: o	47: v	54: 3	61: +
6: G	13: N	20: U	27: b	34: i	41: p	48: w	55: 4	62: /

Padding: =

Sunday, August 11, 2024, 10:33 PM (GMT+0000)

Regular Expressions in C++

In an earlier post I made, I discussed how regular expressions could be used. Now, I will show you how to implement them in your own C++ program.

The `regex` Library

The regular expression library was added in C++ 11, so you should have support for it by now. We can start with some basic boilerplate, importing our regular expression library (along with some other libraries that will make our lives easier) and the standard IO library.

C++

#include <iostream>
#include <string>
#include <vector>
#include <regex>

using namespace std;

int main() {
  return 0;
}

Compiling a Regular Expression

In C++, regular expressions must be compiled before they are used. When I say I am going to be passing a regex string or a regex to a function, I am actually going to be talking about this compiled expression. All regexes must be compiled before use.

It is actually very easy to compile regexes, and below, we are compiling regex <html>.+</html> and assigning the compiled expression to a variable called re.

Then, we will do the same thing I mentioned above but naming the expression reg. The reason I am doing this twice is because I want to show the two methods you can use, assigning by value or assigning using the regex class’s constructor.

C++

#include <iostream>
#include <string>
#include <vector>
#include <regex>

using namespace std;

int main() {
  cout << "Compiling regex 1..." << endl;
  regex re = regex("<html>.+</html>");
  cout << "Compiled regex 1!" << endl;
  
  cout << "Compiling regex 2..." << endl;
  regex reg("<html>.+</html>");
  cout << "Compiled regex 2!" << endl;

  return 0;
}

Determining if a Regular Expression Matches an Entire String

The regex_match function will determine whether an entire string is matched by a certain regex. For example, if we pass the regex hi to it and match it with the string hi, the function will return true, as the regular expression provided matches the entire target string of hi.

However, if we kept the regex the same but changed the target string to shi, the function would return false because while shi contains hi, the regex hi does not match the entirety of shi.

Let’s use an example. I have given one below.

C++

#include <iostream>
#include <string>
#include <vector>
#include <regex>

using namespace std;

int main() {
  string reStr;
  cout<<"Enter a regular expression to use the regex_match function on:\n>";
  cin>>reStr;
  
  string target;
  cout<<"Enter a target string to use the regex_match function on:\n>";
  cin>>target;
  
  regex reCompiled = regex(reStr); // Compiling our regex
  
  // Actual matching process
  if (regex_match(target,reCompiled)) {
    cout<<"\nRegex Matched Entirely!\n";
    return 0;
  }
  else {
    cout<<"\nRegex Did Not Match Entirely!\n";
  }

  return -1;
}

A Quick Note: Capturing Groups

Capturing groups in regexes are denoted by parenthesis and are often returned as lists. To make things simpler, let’s use the regex (sub)(mar)(ine). Here we can see that sub, mar, and ine each have their own capturing groups.

Now if we were to use this on the text submarinesubmarine, the regex would match on both submarines separately, so we would get two matches.

Let’s take a closer look at the matches.

These matches would end up having three submatches each due to these capturing groups. If we were to visualize this in hierarchy, we would get the following:

In C++, matching with capturing groups is represented as a list of matches containing lists of each capturing group for each match. For example, if we wanted to get match one, capturing group one, of a list of matches (you will learn about the smatches type in the next section), we would use the code below:

C++

string m1c1 = matches[0][0];

A Quick Note: The `smatches` Type

The smatches type is used for storing a list of strings as regex matches. It is sort of like a vector, but the shape is fixed to either vector<string> without capturing groups, or vector<vector<string>> with capturing groups.

Determining if a Regular Expression Matches any Substrings

Remember how above, I said that the regex_match function only tells you whether the entire string is matched by a regex? Well, if we want to include substrings, it can get a little more complicated (this is coming from someone with a Python background, where we are pampered with the re library).

For this part of the guide, we will be using the regex_search function, which will tell you if

The regex_search function typically takes three to four arguments. Let’s look at the first method of calling it.

For this method, the function takes three parameters and outputs a Boolean. The parameters are below.

Target (std::string) – This is the string you want to match the regex against
Match Results (std::smatch) – This is the variable of type smatch that will store match results. We will not be using it in this example
Regex (std::basic_regex) – This is the compiled regex that the target is being matched against

The function will return true if any substring of the target string matches the regex, and false otherwise.

C++

#include <iostream>
#include <string>
#include <regex>

using namespace std;

int main() {
    string s="this variable is called s";
    smatch m;
    regex e = regex("s");
    if (regex_search(s,m,e) /* Will return true */) {
        cout<<"Matched! (but not always the entire string)"<<endl;
    }

    return 0;
}

We can also call regex_search using another method, whose parameters are listed below. In this method, we are not only telling the user whether the program is

String Begin and String End – Tells the function to only search the substring in between the string beginning and string ending
Match Results – This is the smatch that will store match results
Regex – The compiled regex that will be used to match against the target string

The function will return true using the same conditions I stated in the previous method, but here what we care about is the fact that the match results are being stored.

The code below will print the first match of the regex, check if there are any matches other than the one it returned, and print the first capturing group. It will keep doing this until there are no other matches. I highly recommend you read the comments in the code below for a better understanding of what it’s doing.

C++

#include <iostream>
#include <string>
#include <regex>

using namespace std;

int main() {
  string target = "submarine submarine submarine";
  regex re = regex("(sub)(mar)(ine)");
  smatch m;
  
  string::const_iterator searchFrom = string::const_iterator(target.cbegin());
  
  // Begin iterating
  while (regex_search(searchFrom,target.cend(),m,re)) {
    
    // We don't want to keep returning the same match every time, so the code below will exclude this match from the future iterations 
    searchFrom = m.suffix().first;
    
    // It is important to know that m[0] would return the entire string ("submarine"), so m[1] will return the first capturing group ("sub")
    cout<<"We have got ourselves a match! \""<<m[1].str() /* First capturing group of match */ <<"\"\n";
    
  }  
}

Regular Expression Find and Replace

The regex_replace function will find and replace all sequences that match the regex.

In the example below, we are telling it to replace all words (including the spaces around them) with “and”

We are also giving it three parameters.

Target – The text that will be replaced accordingly
Regex – The compiled regular expression that will be used on the target
Replace With – The text to replace the matches of the regex with against the target

C++

#include <iostream>
#include <string>
#include <regex>

using namespace std;

int main() {
  regex re("([^ ]+)"); // Matches every word
  cout<<"ORIGINAL: this is text\n";
  cout<<regex_replace("this is text",re,"and"); // prints "and and and"
  return 0;
}

You can also use formatters to incorporate exactly what was replaced using the table below.

Formatter	Example	Explanation
$`number` (where “number” is replaced by any positive number less than 100)	$2	Replaced with the match of the numberth capturing sequence that triggered the replace (starting from 1, such that $1 will get the first capturing group, not $0) at runtime Example: Replacing regex matches of “(sub)(.+)” with “2nd CG: $2” using a target string of “submarine” will yield a result of “2nd CG: marine”
$&	$&	A copy of the entire original string, regardless of capturing groups. Example: Replacing regex matches of “(sub)(.+)” with “String: $&” using the same target string above will result in “String: submarine”
$`	$`	Replaced with whatever came before the match at runtime Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$`” will get replaced with “a “
$’	$’	Replaced with whatever came after the match at runtime Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$’” will get replaced with “marine goes underwater”
$$	$$	I wouldn’t call it a formatter exactly; it’s more of an escape sequence. Used when you don’t want the compiler to mistake the literal character “$” with a formatter. Used when you want to literally type “$” as the text to replace, type “$$”

For example, the code below will put the letters “t” and “e” in parenthesis.

C++

// regex_replace example
#include <iostream>
#include <string>
#include <regex>

using namespace std;

int main ()
{
    regex re("([te])"); // Matches either "t" or "e"
    cout<<"ORIGINAL: thetechmaker.com\n";
    cout<<regex_replace("thetechmaker.com",re,"($&)"); // Prints "(t)h(e)(t)(e)chmak(e)r.com"
    return 0;
}

Monday, June 24, 2024, 5:58 PM (GMT+0000)

Make a Price Drop Notifier in Python
In this guide, I will show you how to use the BeautifulSoup library to make a simple program that notifies you when a product on an online site drops in price.

This library runs in the background, scraping static online e-commerce sites of your choice and notifying you when a product drops in price.

Prerequisites

This guide assumes that you have Python installed, pip added to your system’s PATH, along with a basic understanding of Python and HTML.

Installing Required Components

First, let’s install BeautifulSoup and Requests. The Requests library retrieves our data, but the BeautifulSoup library actually analyzes our data.

We can install those two required components by running the command below:
BAT (Batchfile)
```
pip install beautifulsoup4 requests
```
Note that depending on what your system’s setup is, you might need to use pip3 instead of pip.

Grabbing Our Sample: Price

In this step, we will be telling BeautifulSoup what exactly to scrape. In this case, it’s the price. But we need to tell BeautifulSoup where the price is on the website.

To do this, navigate to the product you want to scrape. For this guide, I will be scraping an AV channel receiver I found on Amazon.

Then, use your browser’s DevTools and navigate to the price. However, make sure that you have a very “unique” element selected. This is an element that shows the product’s price but is also very specifically identified within the HTML document. Ideally, choose an element with an id attribute, as there cannot be two elements with the same HTML ID. Try to get as much “uniqueness” as you can because this will make the parsing easier.

The elements I have selected above are not the most “unique” but are the closest we can get as they have lots of classes that I can safely assume not many other elements have all of.

We also want to ensure that our web scraper stays as consistent as possible with website changes.

If you also don’t have an element that is completely “unique”, then I suggest using the Console tab and JavaScript DOM to see how many other elements have those attributes.

Like, in this case, I am trying to see whether the element I selected is “unique” enough to be selected by its class.

In this case, there is only one other element that I need to worry about, which I think is good enough.

Basic Scraping: Setup

This section will detail the fundamentals of web scraping only. We will add more features as this guide goes on, building upon the code we will write now.

First, we need to import the libraries we will be using.
Python
```
import requests as rq
from bs4 import BeautifulSoup
```
Then, we need to retrieve the content from our product. I will be using this AV receiver as an example.
Python
```
request = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/")
```
If the content you want to scrape is locked behind a login screen, chances are you need to provide basic HTTP authentication to the site. Luckily, the Requests library has support for this. If you need authentication, add the auth parameter to the get method above, and make it a tuple that follows the format of ('username','password').

For example, if Amazon required us to use HTTP basic authentication, we would declare our request variable like the one below:
Python
```
request = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/", auth=("replaceWithUsername","replaceWithPwd"))
```
If that authentication type does not work, then the site may be using HTTP Digest authentication.

To authenticate with Digest, you will need to import HTTPDigestAuth from Request’s sub-library, auth. Then it’s as simple as passing that object into the auth parameter.
Python
```
from requests.auth import HTTPDigestAuth
request = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/", auth=HTTPDigestAuth("replaceWithUsername","replaceWithPwd"))
```
If the content you want to scrape requires a login other than basic HTTP authentication or Digest authentication, consult this guide for other types of authentications.

Amazon does not require any authentication, so our code will work providing none.

Now, we need to create a BeautifulSoup object and pass in our website’s response to the object.
Python
```
parser = BeautifulSoup(request.content, 'html.parser')
```
When you use the Requests library to print a response to the console, you generally will want to use request.text. However, since we don’t need to worry about decoding the response into printable text, it is considered better practice to return the raw bytes with request.content.

Basic Scraping: Searching Elements

Now we can get to the fun part! We will find the price element using our sample we got earlier.

I will cover two of the most common scenarios, one where you need to find the price based on its element’s ID – the simplest, or one where you need to find the price based on class names and sub-elements – a little more complicated but not too difficult, assuming you have a “unique” enough element.

If we wanted to refer to an element based on its ID with BeautifulSoup, you would use the find method. For example, if we wanted to store the element with the ID of pricevalue within a variable called priceElement, we would invoke find() with the argument of id set to the value "pricevalue".
Python
```
priceElement = parser.find(id="pricevalue")
```
We can even print our element to the console!
Python
```
print(priceElement.prettify())
```
Expected Output (may vary)
```
<div id="pricevalue"><p>$19.99</p></div>
```
The function prettify is used to reformat (“pretty-print”) the output. It is used when you want to be able to visualize the data, as it results in better-looking output to the console.

Now we get to the tougher part – making references to element(s) based on one or more class names. This is the method you will need to use for most major e-commerce sites like Amazon or Ebay.

This time, we will be using the find_all function. It is used only in situations where it is theoretically possible to get multiple outputs, like when we have multiple classes as the function gives the output as a list of strings, not a single string.

If you are not sure, know that you can use find_all even when the query you give it only returns one result, you will just get a one item list.

The code below will return any elements with the classes of priceToPay or big-text.
Python
```
priceElements = parser.find_all(class_=["priceToPay","big-text"])
```
The select function is just like that of the find function except instead of directly specifying attributes using its function parameters, you simply pass in a CSS selector and get a list of matching element(s) back.

The code above selects all elements with the class of both price-value and main-color. Although many use the find or find_all functions, I prefer select as I am already familiar with CSS selectors.

If, and this is not much of a good idea when finding elements, we would like to filter by element type, we will just call find_all with a single positional argument, the element’s type. So, parser.find_all("p") will return a list of every single paragraph (“p“) element.

An element type is one of the broadest filters you can pass into the find_all function, so this only becomes useful when you combine it with another narrower filter, such as an id or class.
Python
```
parser.find_all("h1", id="title")
```
That would return all h1 elements with an ID of title. But since each element needs to have its own unique ID, we can just use the find function. Let’s do something more realistic.
Python
```
parser.find_all("h1",class_="bigText")
```
This code would return all h1 elements that had a class of bigText.

Below are a few reviews of what we know so far and some other, rarer methods of element finding.
Python
```
"""
Never recommended, but returns a list of ALL the elements that have type 'p'
"""
typeMatch = parser.find_all("p")

"""
Finds element with the ID of 'priceValue' using a CSS selector
"""
idSelMatch = parser.select("#priceValue")

"""
Finds element with the ID of 'priceValue', except with the BeautifulSoup-native find function and not with a CSS selector
"""
idMatch = parser.find(id="priceValue") # Same as above


"""
Extremely rare, but returns a list of elements containing an ID of 'priceValue' OR 'price'
"""
orIdMatch = parser.find_all(id=["priceValue","price"])


"""
Returns a list of elements that have the class 'price' OR 'dollarsToPay'. I do not know of a CSS selector that does the same
"""
orClassMatch = parser.find_all(class_=['price','dollarsToPay'])


"""
Returns a list of elements that have the class 'price' AND 'dollarsToPay'. I do not know of a 
find_all argument that does the same
"""
andClassMatch = parser.select(".priceValue.dollarsToPay")

"""
Returns the element that has a class of 'v' INSIDE the element of class 't'. This can also be done with ID attributes, but this function only works when the first function is .find(...) or when you are grabbing an element by index after calling .find_all(...). Because .find(...) only returns one element, it will only be returning the first instance of that class name. The code below return the same thing, however 'inMatch3' returns a list
"""
inMatch = parser.find(class_="t").find(class_="v") # Most basic way to do it
inMatch2 = parser.find_all(class_="t")[0].find_all(class_="v")[0] # Because .find_all(...) works on the final element, the '[0]' is unnecessary, we just do it so we don't get a one-element list
inMatch3 = parser.find_all(class_="t")[0].find_all(class_="v") # Returns a one-element list
```
Now that we know how to search elements, we can finally implement this in our price drop notifier!

Let’s see if our request is successful. We will be printing out the entire file to check.
Python
```
print(parser.find("html").prettify())
```
And we are not.

Hmmm, so we have to bypass Amazon’s CAPTCHA somehow, so let’s try adding headers that mimic a normal browser!

I will be adding headers to rq.get(). Make sure to replace my AV channel receiver link with the product you want to scrape.
Replace “request=rq.get(…)”
```
headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","Sec-Ch-Ua":'"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',"Sec-Ch-Ua-Mobile":"?0","Sec-Ch-Ua-Platform":"\"Windows\""}

request = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/",headers=headers)
```
Let’s try now…

Nope. Still nothing. Well, time for plan B, ditching requests completely and using selenium.

Sign up for our newsletter!

Name

Email

By submitting your information, you’re giving us permission to email you. You may unsubscribe at any time.

Δ

Basic Scraping: Implementation of Selenium

Firstly, it is important to know that Selenium has its own methods for finding elements in a HTML document, but for the sake of this guide, we will just be passing the source code of the website to our parser.

Think of Selenium as a browser running in the background with some selection abilities. Instead of sending the requests to the website by crafting our own headers, we can use Selenium to spin up an invisible browser that crafts the headers for us. We should no longer get a CAPTCHA screen because Amazon shouldn’t be suspicious that a robot is browsing the page – we are technically using a legitimate browser, but with parsing capabilities.

Installation of Selenium can be done with the command below. We will also be installing win10toast so you get a proper toast notification whenever a price drop is detected.
BAT (Batchfile)
```
pip install selenium
pip install win10toast
```
If you are looking for how you can uninstall Requests because you don’t need it anymore, think twice because Selenium depends on Requests anyways.

Now, clear your entire Python file because we are going to need to do a short and quick rewrite of our code to use Selenium.

Like always, we will start by importing the required modules. Make sure you replace chrome with the name of a browser you have installed on your system, preferably the most resource efficient one.
Python
```
from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options # Imports the module we will use to change the settings for our browser
import time # This is what we will use to set delays so we don't use too many system resources
from win10toast import ToastNotifier # This is what we will use to notify if a price drop occurs.

notifier = ToastNotifier() # Assign our notifier class to a variable
```
Then, we will need to set some preferences for the browser we are about to start. Let’s start by declaring an Options class and using it to make the browser invisible or run it in “headless” mode. While the arguments below are for specific browsers, I would just execute them all because I have not tested each argument individually.
Python
```
browserOptions = Options()
browserOptions.headless = True # Makes Firefox run headless
browserOptions.add_argument("--headless=new") # Makes newer versions of Chrome run headless
browserOptions.add_argument("--headless") # Makes older versions of Chrome run headless
browserOptions.add_argument("--log-level=3") # Only log fatal errors
```
Now, we will initiate the browser in the background. Again, make sure you replace Chrome with whichever browser you want to use for this project.
Python
```
browser = webdriver.Chrome(options=browserOptions)
```
Now, we can navigate our browser to the page we want to scrape and get its source, which we can pass to BeautifulSoup.
Python
```
browser.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/")
parser = BeautifulSoup(browser.page_source, "html.parser")
```
Then, we can use what we already know about BeautifulSoup to grab the price of our element. Remember to replace the code below with one tailored to your sample.
Python
```
price = parser.select(".a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay")[0].find_all(class_="a-offscreen")[0].text
```
Next, let’s strip the $ symbol from the price and convert it into a floating-point decimal.
Python
```
price = float(price.strip("$"))
```
Then, we can set a variable to compare with the current price.
Python
```
previousPrice = price
```
Now, we loop infinitely to see whether the price changed.
Python
```
while True:
```
Insert a new line and then indent the code we will write from this point forward.

Now, every two minutes (120 seconds), we refresh the page and compare the price we just got to our previous price.
Python (place each line indented inside while loop)
```
browser.refresh() # Refreshes the browser

# Now that we may have a new price, we have to redfine our parser and price variables to adapt to that new page code
parser = BeautifulSoup(browser.page_source, "html.parser")
price = parser.select(".a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay")[0].find_all(class_="a-offscreen")[0].text
price = float(price.strip("$"))

# Next, we compare the two prices. If we find one, we alert the user and update our price threshold. We will also be looking for price increases.
if (price<previousPrice):
  print(f"Price DECREASED from ${previousPrice} to ${price}!")
  notifier.show_toast("Price Drop!", f"The price decreased from ${previousPrice} to ${price}!")
elif (price>previousPrice):
  print(f"Price INCREASED from ${previousPrice} to ${price}!")
  notifier.show_toast(":(", f"The price increased from ${previousPrice} to ${price} :(")

# Now, we can tell the user we refreshed
print(f"Refreshed! Previous Price: ${previousPrice}, and new price ${price}")
previousPrice = price

# And then we wait for two minutes
time.sleep(120)
```
And just like that, you are finished! I hoped this project was useful to you!
Thursday, February 1, 2024, 9:18 PM (GMT+0000)
The Beginner’s Guide To Regular Expressions

Regular expressions (abbreviated as a “regex”) are useful tools that help easily find and match text in strings and files. They sort of function like the typical find-in-document feature you are probably used to, only more advanced. However, they can definitely be too advanced for an average person to comprehend. In this guide, I will easily explain this widely used and useful feature so you can search files like a pro.

Something Worth Noting

It is worth noting that regexes are very useful and have lots of features to utilize, but can also be very non-standardized. There are always features in other regex parsing engines that are not supported by others. The features I will be displaying in this guide are the most widely supported ones, but there are always going to be features that I won’t teach you how to use here either because they are not known or because they are defunct and not considered good practice.

Basic Matching

Regexes can match specific strings, no complex syntax needed. For example, to match the text foo in notnotnotfoonotnot, your regex would simply be foo.

This works with multiple instances too; the regex foo would match both instances of foo in notnotnotfoonotnotfoo.

Something else that you should know about basic matching is that there will be two seperate matches if the same matching string is used multiple times (together or not).

This means that the regex foo would match all three foo instances in notnotfoofoonotfoo (notice how there are separate matches for the two instances of foo, even though said instances are together).

Match Any Character

The period selects any character. Note that each character is matched separately.

For example, the regex . in the text foo will match f, o, and the second o.

Notice how in the image above, even special characters and numbers are matched.

Letter Range Match

Let’s say in the text abcdefghijklmnopqrstuvwxyz, we want to match just e, f, g, h, and i. To do this, we can select the letter range that we want (in this case, e through i) using the regex [e-i].

Note that these letter ranges do not have to be subsequent; we can have the regex [a-c] in the text azaczbca match a, a, c, b, c, and a.

Number Range Match

Let’s say in the text 019201836y7 you only want to select the numbers 0, 1, 2, the second 0, and the second 1. In other words, we want to select any number from zero to two, including themselves. We can do this using the number range match, and in this case, we can use the regex [0-2].

These letter ranges also don’t have to be consecutive.

Character Set

Now let’s say you want to match specific characters that can’t be matched with a set. For this, we will use a character set. For example, we can match car and bar in the text car,bar using a character set. Because only the beginning letters of each word change, we can use a character set to match the c and b, and then use basic matching to match the ars. To do this, we can put in the characters we want to put in the set in square brackets.

Ignored Character Set

How about if we wanted to match every character except for c and b. To do this we can use an ignored character set to get the job done. If we wanted to match only tar in the text car,bar,tar, we can use an ignored character set to ignore c and b, then use basic matching as all three words have different initials but the same of everything else. To do this, we can put the characters we want to ignore in square brackets with the character ^ coming before them.

The Asterisk

The asterisk (*), when placed outside a character indicates that said character should be matched even if it does not occur at all or occurs at least once side by side. For example, the regex a* will match at least zero of a, so aaaa will be matched in the text aaaab. This is different from basic matching, as the regex a will count every a as a separate match, while a* will keep all the as in one match as many times as needed to complete the match.

Another examples is that the asterisk is like saying the character that comes before it is optional or can occur multiple times in a row. The regex ab*c in the text abc,abbc,ac will match abc, abbc, and ac.

The Question Mark

The question mark is just like the asterisk in the sense that the character that comes before it is deemed optional, but different in the sense that it will not match characters that occur multiple times in a row. For example, the regex ab?c will match abc and ac, but not abbc.

The Plus Symbol

The plus symbol is also just like the asterisk in the sense that the character that comes before it can occur multiple times in a row, but different in the sense that it does not indicate a character is optional. If we go back to our abc example, we can see that the regex ab+c in abc,abbc,ac will match abc and abbc, but not ac.

Conclusion

And that’s it – you’re done! There are so many other regex sequences you can use, and they just won’t fit into a beginner’s guide. As you saw above, regular expressions are very useful tools, and there are many ways to apply this to your needs.

Sunday, November 19, 2023, 8:52 PM (GMT+0000)
How to Install Arch Linux From Scratch
In this guide, I will be showing you how to install Arch Linux from scratch. You can do this on a physical machine, but I will be doing it on a virtual machine.

What is Arch Linux?

Arch Linux is a minimal Linux distro that is meant for power users and advanced programmers. It does not come with a built-in installer, so we will have to install it manually.

Downloading the ISO

First, head to the Arch Linux downloads page, scroll down until you find the download mirrors, and then choose a link to download, preferably from your country so you get the fastest download speed.

VM Configuration

If you are installing Arch Linux on physical hardware and you are not using a VM to install Arch, skip this section.

If you are on VirtualBox, there should be an Arch Linux preset. If you are on VMware, select Other Linux 5.x Kernel (64-bit).

Giving your VM 8 GB of RAM is a lot more than needed, but if you are going to be using Arch for power-intensive tasks and don’t mind the VM taking up all your host’s RAM, go for whatever fits for you as long as it meets the system requirements of 512 MB minimum, but 2 GB recommended for streamlined daily use.

Now, give your VM any amount of storage you feel fitting, but make sure it meets the system requirements of 1 GB minimum, but 20 GB recommended.

If you want things to go a little faster on your Arch Linux VM, giving it two processors is recommended. One processor should be enough, though.

If you plan on installing a desktop environment, enable 3D acceleration and give a reasonable amount of VRAM to the guest OS.

And lastly, make sure the CD drive is set to read as the ISO you just downloaded.

Making a Bootable USB

If you are using a VM, skip this section.

Use a tool like Rufus to flash the Arch Linux ISO to a flash drive. Then plug it into the system you want to install Arch Linux to. This guide does not cover dual-booting and assumes you do not have an existing OS installed on your system.

Boot Priority Configuration

Make sure the hard disk is the first on the boot priority list. This is not required; it just makes the final step of the installation a lot quicker.

You can do this in the VM’s settings, the UEFI firmware settings, or the BIOS interface.

Starting the System

If you are on a VM, start it up. If you are on hardware, plug in the thumb drive that you flashed the ISO to and start the machine.

You should see an Arch Linux splash screen. Whichever entry comes first in the list is likely the one that boots to Arch (the name of the entry changes, but typically goes along the lines of “Boot Arch Linux (x86_64)”). Select Boot Arch Linux, or the entry that does so, and click Enter.

After running some tests, you should be dropped into a root Linux shell. Do not remove your thumb drive or installation media, as we have not installed Arch yet and need to do so using the thumb drive.

Accessing the Internet

Let’s check if we have connection to the internet by running the command below (“google.com” can be replaced with a website of your choice):
Bash
```
ping google.com
```
You should see packets return. If you do, this means you have online access, which is a necessity for this installation. You can hit Control + C to stop pinging the website.

If you are on a Virtual Machine with no internet, try making sure that you have host internet access, and then try enabling the network adapter in the VMware or VirtualBox settings.

If you are on a physical machine without internet, try using an Ethernet cable, but below you will know how to use iwctl to connect to a Wi-Fi network.

Using iwctl to Connect to a Wi-Fi Network

This part of the guide should only be followed if you do not have Internet access and want to use a wireless internet connection instead of a wired one.

Run the command below to open the iNet Wireless Daemon CTL.
Bash
```
iwctl
```
Expected Output
```
[iwd]# 
```
Now, you can list the wireless adapters using the command below.
Bash
```
device list
```
You should see a device called wlan0. It is best to go forward using that one, but you can select another wireless adapter if you know what you are doing. If you do not have that device, then you do not have a wireless adapter plugged into the computer, the device is connected but under a different name (which is unlikely), or the computer does not recognize it.

Use the command below to list all the wireless networks found using that device. You can replace wlan0 with the adapter you chose earlier.
Bash
```
station wlan0 get-networks
```
You should see a list of networks. Take note of the network’s SSID you want to connect to. Then, you can run the command below to connect to the wireless network, replacing “WirelessNet” with the SSID of your wireless network and wlan0 with the wireless adapter you want to connect to the network using.
Bash
```
station wlan0 connect WirelessNet
```
After typing in the Wi-Fi password (if needed), you may now connect to the network. You can test your connection by using ping google.com and waiting for packets to return. If none return, then you might have done something incorrectly when setting up the network.

Setting NTP Time

Now that we have our network up and running, we can enable network time using the command below:
Bash
```
timedatectl set-ntp true
```
Partitioning

Now comes the tricky part. We have to partition our drives manually, so make sure to follow my steps carefully.

Run the below command to get a summary of all the partitions on your drive.
Bash
```
lsblk
```
Make sure you choose the right disk to partition, as choosing the wrong one will destroy all of your data. Run the below command to set up partitioning, replacing /dev/sda with the name of the disk you want to format and install Arch Linux on.

As for the label type, it depends what your needs are. If you are installing to a new physical system with a disk size larger than 2TB, select gpt and hit Enter. If you either don’t have a physical system to install Arch to or are installing Arch to a disk smaller than 2TB, use dos. Now, at the menu with drives listed, select the free space and click New. When asked for your partition size, enter the amount needed for your bootloader. If you will be using the GRUB bootloader, enter 128M and hit Enter. If not, specify the amount needed for your bootloader.

Now, select the newly created partition and hit B to make the selected drive bootable.

Select the free space, and click New. The size should automatically default to the remaining storage on your drive. Make the partition fill up the rest of the drive, and click Enter to create the partition. You should not make this partition bootable.

Many people prefer creating home and swap partitions but these are mostly considered redundant nowdays.

Select Write and click Enter, then select Quit and click Enter.

There is only one problem left to solve now – the drives are not in the format of ext4. To solve this, run the commands below, replacing /dev/sda1 and /dev/sda2 with the names your newly created boot and OS partitions.
Bash
```
mkfs.ext4 /dev/sda1
mkfs.ext4 /dev/sda2
```
Mounting our Partitions

Now that we are done with arguably the hardest part of Arch installation, we need to mount our drives, which is where the preparation ends and the actual installation starts.

To begin, lets mount our root partition (the partition that is not bootable, /dev/sda2 in my case) to a mount point (this can be anything you want, but traditionally this has always been /mnt. I will be using /mnt, as I do not see any reason to stray from tradition in my case). We can do this using the command below.
Bash
```
mount /dev/sda2 /mnt
```
In our mount point, let’s create a folder called boot to mount our other drive to.
Bash
```
mkdir /mnt/boot
```
Now, let’s mount our boot partition (the one that we flagged bootable earlier, /dev/sda1 in my case) to the folder we just created.
Bash
```
mount /dev/sda1 /mnt/boot
```
To see if we did everything correctly, we can run the command below.
Bash
```
lsblk
```
In the output, you should be able to see partitions under the drive /dev/sda and their respective mount points.

Installing Arch Using pacstrap

Now, we can begin installing system files to Arch.

We can use pacstrap to install Arch Linux and some other packages we want pacstrap to bundle in with our Arch installation. Replace /mnt with the mount point you mounted your root drive to, and vim with some other text editor that you prefer and some other pre-installed packages you want on Arch.
Bash
```
pacstrap /mnt base base-devel linux linux-firmware vim
```
Below ae explanations of some of the packages:
- Base: This package contains basic tools that you would want no matter which Linux distribution you are installing.
- Base-Devel: This contains developer tools such as compilers, which are needed for some Linux components.
- Linux: This is the core Linux kernel that runs everything behind the scenes.
- Linux-Firmware: This contains firmware that makes Arch compatible with common hardware
- Vim: This can be replaced with any text editor. There is some text editing we are going to have to do, so we need a text editor.
Once pacstrap exits, we can generate an fstab file that lists all the drives Linux could try booting from.

Generating our FSTab File

This is extremely easy to do. Run the command below, replacing /mnt with the mount point you specified earlier.
Bash
```
genfstab -U /mnt >> /mnt/etc/fstab
```
What the command should do is write a FSTab file to the drives. The -U flag makes the file reference drives by UUID instead of drive name. This is good because drive names may change in the future, which could mess up your boot configuration if you don’t reference drives by UUID, as UUID never changes.

Jumping to our Arch Installation

It is finally time to change from our installation medium to our disk. Do not remove your installation medium until the end of this guide. You might need it if something breaks.
Bash
```
arch-chroot /mnt /bin/bash
```
After this command, do not reboot yet. We still have to install our boot manager.

Installing Basic Components

Now that we are in Arch, we have access to the Pacman package manager. We can use it to install basic components like a network manager for accessing the internet, and a boot manager so we can boot into the system. I will be installing GRUB as a boot manager, but you can install something else.

Use the command below to install these components.
Bash
```
pacman -S networkmanager grub
```
Configuring the Network Manager to Run on Startup

If we want internet at boot, we are going to have to enable NetworkManager’s system service. we can do this using the command below.
Bash
```
systemctl enable NetworkManager
```
Configuring the GRUB Boot Manager

Now, we have to configure what our system boots using. When we ran the Pacman command, we downloaded GRUB, but we did not install or configure it. Let’s install GRUB using the command below, replacing /dev/sda with whatever drive you are using to install Arch. We are not going to be using /dev/sda1 or /dev/sda2 because it is critical that you install it to the drive, not the drive partition.
Bash
```
grub-install /dev/sda
```
Now, we can make a GRUB configuration file using the command below.
Bash
```
grub-mkconfig -o /boot/grub/grub.cfg
```
Take a look at the output and make sure that it says both an initrd image and a Linux image were found. If it does not find these images, the most likely cause is incorrectly installing the Linux kernel using pacstrap.

Setting a Password

Now, run the command below to create a password for your root user.
Bash
```
passwd
```
Configuring Locales and Timezones

Use the command below to enter the locale configuration file, replacing vim with whatever text editor you installed earlier.
Bash
```
vim /etc/locale.gen
```
Now, use the arrow keys or the scroll wheel to scroll down to the locale you want to use. I will be using United States English, so I will scroll down to en_US and uncomment (remove the # before) both the UTF and ISO formats. If you are using Vim, you might have to hit the I key on your keyboard before you can actually type anything.

Write the file and exit your text editor. To write and exit Vim, we can hit the Escape key on our keyboard and type :wq.

Now that we have our locales configured, we have to apply the changes by generating the locales. We can do this using the command below.
Bash
```
locale-gen
```
Now, we also have to create a file called locale.conf to define our language in. Use the command below, once again replacing vim with your desired text editor.
Bash
```
vim /etc/locale.conf
```
In the file, type LANG=en-US.UTF-8, once again replacing en-US with whatever locale you are using. Exit Vim.

Now that we have the timezones prepared, we have to link them to make our system clock show the right timezone. Type ln -sf /usr/share/zoneinfo and click Tab. This will list all the broadest timezones. Keep making the directories more specific, hitting Tab to see the available options every time, and after you are done typing a city, hit Space and type /etc/localtime.

Setting Our Hostname

Now, we can set our hostname. This is the name that the Arch machine will use to communicate with other devices over your network. By default, your hostname is archiso. If you are happy with that and don’t want to change it, you can skip this section.

Use your prefered text editor to create /etc/hostname. Do not include a file extension. Type whatever you want your system hostname to be, and exit.

Unmounting Our Disk

Now, we can exit our chroot jail by using the command below.
Bash
```
exit
```
Now would be a good time to check and make sure your hard drive is first boot priority. Make sure that when you return, you are in your installation medium and not in the actual Arch installation.

Unmount our Arch installation with the command below, replacing /mnt with the mount point you specified earlier.
Bash
```
umount -R /mnt
```
Now, we can boot into our new installation of Arch Linux using the command below. Once you have booted in, you may remove the installation medium.
Bash
```
reboot
```
Sunday, October 1, 2023, 12:57 AM (GMT+0000)
Optimizing AI Models Using Convolutional Neural Networks
This guide is a part two to a previous guide I made, called The Simple Guide to AI and Machine Learning With Python. This guide is simply how you can improve accuracy to the model you made in that guide, meaning that I’m going to assume you have already completed the previous guide before going on to follow this guide.

In the previous guide, we learned how you can use dense neural networks to make a program that recognizes handwriting. Well, that neural network was not exactly very accurate, as it had a tendency to get numbers wrong unless it was specifically modified for those numbers. As you probably know by now, you would probably want the neural network to recognize any number you give it without having to optimize the network for every single number that comes to it.

Convolutional neural networks were made to solve this problem. Rather than training off of the overall image, convolutional neural networks recognize tiny features in the image and learns those. For example, rather than focusing on the entire image of a hand-drawn three, the network will learn that a three has two curves that are stacked vertically, which will help it recognize any other threes in the future, no matter how it was drawn or whether the neural network was optimized for the number three.

Step One: Initial Setup

For this step, we can just use the code that we used in the previous tutorial to prepare the MNIST dataset.
Python
```
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras import backend as K
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# helper functions
def show_min_max(array, i):
  random_image = array[i]
  print("min and max value in image: ", random_image.min(), random_image.max())


def plot_image(array, i, labels):
  plt.imshow(np.squeeze(array[i]))
  plt.title(" Digit " + str(labels[i]))
  plt.xticks([])
  plt.yticks([])
  plt.show()

def predict_image(model, x):
  x = x.astype('float32')
  x = x / 255.0

  x = np.expand_dims(x, axis=0)

  image_predict = model.predict(x, verbose=0)
  print("Predicted Label: ", np.argmax(image_predict))

  plt.imshow(np.squeeze(x))
  plt.xticks([])
  plt.yticks([])
  plt.show()
  return image_predict

img_rows, img_cols = 28, 28  

num_classes = 10 

(train_images, train_labels), (test_images, test_labels) = mnist.load_data() 
(train_images_backup, train_labels_backup), (test_images_backup, test_labels_backup) = mnist.load_data() 

print(train_images.shape) 
print(test_images.shape) 

train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols, 1)
test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

train_images = train_images.astype('float32')
test_images = test_images.astype('float32')

train_images /= 255
test_images /= 255

train_labels = keras.utils.to_categorical(train_labels, num_classes)
test_labels = keras.utils.to_categorical(test_labels, num_classes)

print(train_images[1232].shape)
```
Expected Output
```
(60000, 28, 28)
(10000, 28, 28)
(28, 28, 1)
```
Now that we have already put in the initial setup of our code, we can jump straight to creating our network.

Creating Our Network

Similar to what we did with the densely connected network, we are still going to have epochs, or the amount of times the network goes through the entire set over again.

With that explanation out of the way, we can define our model.
Python
```
from tensorflow.keras.models import Sequential 
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropout

epochs =  10
model = Sequential()
```
Now, let’s start adding the layers of our neural network.

Explaining Convolutional Layers

With our previous network, we added three dense (fully connected) layers. With our new network that uses convolutional neural networks, the layers work differently.

Convolutional layers consist of groups of neurons called filters that move across the image and activate based on the pixels they read. Those groups will then learn how to recognize features in the data.

It is possible to adjust amount and size for filters in your neural network, which we will change to our liking. Bigger filters can observe larger parts of the image at once, while smaller filters gather finer details about the image. A higher amount of filters means that the neural network can recognize a wider range of image features.

There are many advantages of having layers and filters work this way. For one thing, smaller filters can be more computationally efficient by only examining a small part of the image at once. Furthermore, as filters are moved across the entire image, the neural network will not be affected by feature displacement (occurs when a feature is common to two images, but in different spots of an image). Just like reality, filters focus on a small area of the image, so they are not distracted by the other parts of an image.

We will be using multiple convolutional layers to complete our new-and-improved handwriting recognition software.

Implementing Convolutional Layers

When we use Keras, we can easily take advantage of its functionality to easily create convolutional layers that we will then use in our model. We will use the Conv2D function to create the first layer of out neural network.

In the case below, we will have 32 filters, a kernel size of (3,3), an input shape – which we saved to the input_shape variable when we ran the setup code at the beginning – of (28,28,1), and an activation function of ReLU. I go more in-depth into what ReLU is in my previous guide.
Python
```
model.add(Conv2D(filters=32, kernel_size=(3,3),activation='relu',input_shape=input_shape))
```
The Conv2D function creates 2D convolutional layers, meaning that they scan across flat data, like images.

Explaining Pooling Layers

When you use convolutional layers, things can get quite computationally intensive, which is where pooling layers come in. Increasing the number of neurons will increase the number of computation time required. Pooling layers are essentially filters that move in specified strides across the image, simplifying each of the filters’ contents into a single value. This, based on the size and stride of the filter, shrinks the output image.

For this scenario, we will have a 2×2 filter with a size of 2. This halves the image’s row and column count, simplifying the data without too much loss of specificity.
Python
```
model.add(MaxPooling2D(pool_size=(2,2)))
```
Most networks have at least one set of alternating convolutional and pooling layers.

More Convolutional Layers

Convolutional layers are designed to examine the low-level features of an image. If we add more, we may be able to start working with higher-level features.

We define the layer the same way we defined the previous one, but now we have 64 filters, not 32. We also do not specify the input shape, as it is inferred from the previous layer.
Python
```
model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu'))
```
Dropout Layers

Dropout layers are layers that take a percentage of all input neurons and deactivate them randomly. This forces other neurons to adapt to the task. When larger and more complicated networks lack a dropout layer, the network risks being too dependent on a single set of neurons rather than all neurons learning. This is called Overfitting and can change your network output for the worse.

Below, we will have our dropout layer take 30%, or 0.3 neurons to deactivate randomly.
Python
```
model.add(Dropout(rate=0.3))
```
Dense and Flatten Layers

After all the convolutional and pooling layers, we will need a layer to help make our final decision. This will be a regular, fully connected dense layer. Before we connect this layer, we will need to flatten the image’s filters.

We can start by flattening the image using the Keras Flatten layer.
Python
```
model.add(Flatten())
```
Now, we can add a dense layer with ReLU activation and 32 neurons.
Python
```
model.add(Dense(units=32,activation='relu'))
```
Sign up for our newsletter!

Name

Email

By submitting your information, you’re giving us permission to email you. You may unsubscribe at any time.

Δ

Output Layers

Similar to the fully connected neural network we made in the previous guide, we will need a layer to shrink the previous dense layer down to just the number of classes. Also similar to before, the final output is decided by using the class with the highest weight.

Below, we will add a dense layer to be our output layer. The number of neurons should be 10 because there are ten possible output classes, and the activation should use Softmax.
Python
```
model.add(Dense(units=10,activation='softmax'))
```
Model Summary

Now, we can print out our model summary:
Python
```
model.summary()
```
Expected Output (Lines Providing no Useful Data are Blurred)
```
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 26, 26, 32)        320       
                                                                 
 max_pooling2d (MaxPooling2  (None, 13, 13, 32)        0         
 D)                                                              
                                                                 
 conv2d_1 (Conv2D)           (None, 11, 11, 64)        18496     
                                                                 
 dropout (Dropout)           (None, 11, 11, 64)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 9, 9, 32)          18464     
                                                                 
 flatten (Flatten)           (None, 2592)              0         
                                                                 
 dense (Dense)               (None, 32)                82976     
                                                                 
 dense_1 (Dense)             (None, 10)                330       
                                                                 
=================================================================
Total params: 120586 (471.04 KB)
Trainable params: 120586 (471.04 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
```
Compiling and Training

Now we will compile the network. The loss and metric will be the same as the ones that we use in the previous guide, Categorical Cross Entropy and accuracy respectively. However, we will use RMSProp (Root Mean Squared Propagation) as our training algorithm. RMSProp is one of many training algorithms that Keras can use to teach the network how to actually improve, optimizing the loss to make it as small as possible. We will achieve this using RMSProp.
Python
```
model.compile(loss='categorical_crossentropy', optimizer='rmsprop',  metrics=['accuracy'])
```
Now, we can start training.

The fit function is the one that actually does the training.

Now we can look at the parameters of the training function.
- train_images and train_labels state the data that this neural network model will be trained on. The images are the pieces of data given to the network, and the network tries to find out the appropriate label
- batch_size allows us to put the network’s data into batches. We can always change it later, but for now we have set it to 64
- epochs defines the number of epochs (times the network reiterates on the training data) the network should use
- validation_data defines the data the model is testing itself on
- We have turned shuffle on so Keras shuffles the training data after every epoch and isn’t relying on the order of the data to train on
Python
```
model.fit(train_images, train_labels, batch_size=64, epochs=epochs, validation_data=(test_images, test_labels), shuffle=True)
```
Expected Output (may vary)
```
Epoch 1/10
938/938 [==============================] - 23s 24ms/step - loss: 0.1677 - accuracy: 0.9473 - val_loss: 0.0501 - val_accuracy: 0.9832
Epoch 2/10
938/938 [==============================] - 23s 24ms/step - loss: 0.0512 - accuracy: 0.9841 - val_loss: 0.0331 - val_accuracy: 0.9885
Epoch 3/10
938/938 [==============================] - 22s 23ms/step - loss: 0.0354 - accuracy: 0.9894 - val_loss: 0.0347 - val_accuracy: 0.9894
Epoch 4/10
938/938 [==============================] - 22s 24ms/step - loss: 0.0283 - accuracy: 0.9918 - val_loss: 0.0349 - val_accuracy: 0.9879
Epoch 5/10
938/938 [==============================] - 22s 23ms/step - loss: 0.0228 - accuracy: 0.9928 - val_loss: 0.0271 - val_accuracy: 0.9911
Epoch 6/10
938/938 [==============================] - 22s 24ms/step - loss: 0.0199 - accuracy: 0.9938 - val_loss: 0.0273 - val_accuracy: 0.9909
Epoch 7/10
938/938 [==============================] - 22s 23ms/step - loss: 0.0155 - accuracy: 0.9953 - val_loss: 0.0299 - val_accuracy: 0.9904
Epoch 8/10
938/938 [==============================] - 22s 24ms/step - loss: 0.0140 - accuracy: 0.9956 - val_loss: 0.0321 - val_accuracy: 0.9911
Epoch 9/10
938/938 [==============================] - 22s 24ms/step - loss: 0.0120 - accuracy: 0.9960 - val_loss: 0.0387 - val_accuracy: 0.9905
Epoch 10/10
938/938 [==============================] - 23s 25ms/step - loss: 0.0112 - accuracy: 0.9968 - val_loss: 0.0334 - val_accuracy: 0.9918
```
Model Evaluation

Now, we have to test the model on data it hasn’t seen yet. To do this, we will use the evaluate function. Loss and accuracy are percentages returned in decimal format.
Python
```
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
```
Expected Output (may vary)
```
313/313 - 1s - loss: 0.0334 - accuracy: 0.9918 - 886ms/epoch - 3ms/step
```
In our case above, the accuracy was 99.18%, which is pretty good.

Exporting Our Model

Now, we can export the model to be used elsewhere. We can do this by using model.save.
Python
```
model.save('cnn_model.h5')
```
This will save the model to a file called “cnn_model.h5”, where it can then be loaded in other pieces of code.
Sunday, August 20, 2023, 11:28 PM (GMT+0000)
The Simple Guide to AI and Machine Learning With Python
In this guide, you will learn how to create an AI that recognizes handwriting with Python using Dense neural networks and the MNIST dataset. This guide will use TensorFlow to train your AI, and basic knowledge of linear algebra used in AI is strongly recommended. You can refer to this guide to understand the linear algebra used in AI. In the next part, we upgrade the neural network’s accuracy using convolutional neural networks.

Prerequisites

To do this, you will first need to install Python and add Pip to the .bashrc file for Linux or the Environment Variables in Windows or Mac. Then, run the command below to install the required libraries:
BAT (Batchfile)
```
pip install "tensorflow<2.11"
pip install pandas openpyxl numpy matplotlib
```
Note: If installing TensorFlow does not work, you can run pip install tensorflow. This will function like normal, but it will not be able to utilize your GPU.

Writing The Code

In a new Python file, we will first import the dataset and import the libraries needed:
Python
```
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras import backend as K
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.models import Sequential 
from tensorflow.keras.layers import Dense, Flatten
```
We then define some functions that will help us visualize the data better later on in the code. I will not go over how they work, but they are not a necessity, just there to help us visualize the data better:
Python
```
def show_min_max(array, i):
  random_image = array[i]
  print(random_image.min(), random_image.max())

def plot_image(array, i, labels):
  plt.imshow(np.squeeze(array[i]))
  plt.title(" Digit " + str(labels[i]))
  plt.xticks([])
  plt.yticks([])
  plt.show()
  
def predict_image(model, x):
  x = x.astype('float32')
  x = x / 255.0

  x = np.expand_dims(x, axis=0)

  image_predict = model.predict(x, verbose=0)
  print("Predicted Label: ", np.argmax(image_predict))

  plt.imshow(np.squeeze(x))
  plt.xticks([])
  plt.yticks([])
  plt.show()
  return image_predict
  

def plot_value_array(predictions_array, true_label, h):
  plt.grid(False)
  plt.xticks(range(10))
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array[0], color="#777777")
  plt.ylim([(-1*h), h])
  predicted_label = np.argmax(predictions_array)
  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')
  plt.show()
```
In the MNIST Data set (the dataset that we will be using), there are 60,000 training images and 10,000 test images. Each image is 28 x 28 pixels. There are 10 possible outputs (or to be more technical, output classes), and there is one color channel, meaning that each image is stored as a 28 x 28 grid of numbers between 0 and 255. It also means that each image is monochrome.

We can use this data to set some variables:
Python
```
img_rows = 28 # Rows in each image
img_cols = 28 # Columns in each image
num_classes = 10 # Output Classes
```
Now, we will load the train images and labels and load in another set of images and labels used for evaluating the model’s performance after we train it (these are called test images/labels).

What Are Images and Labels?

These can also be data and labels. The data is the context that the computer is given, while the labels are the correct answer to predicting based on data. Most of the time, the model tries predicting labels based on the data it is given.
Python
```
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
```
The next step is not required, and we don’t make use of it throughout the code, however it is recommended, especially if you are using a Python notebook.

The next step is to create a duplicate, untouched version of the train and test data as a backup:
Python
```
(train_images_backup, train_labels_backup), (test_images_backup, test_labels_backup) = mnist.load_data()
```
Now, we test to see if we loaded the data correctly:
Python
```
print((train_images.shape, test_images.shape))
```
Expected Output
```
((60000, 28, 28), (10000, 28, 28))
```
Why Are They Those Shapes?

The images are 28×28, so that explains the last two dimensions in the shape. Because the data is stored as a long matrix of pixel values (this is not readable to our neural network, by the way; we will fix this later), we do not need to add any more dimensions. If you remember what I said earlier, you will know that there are 60000 training images and 10000 testing images, so that explains the first dimension in the tensor.

The whole purpose of this tutorial is to get you comfortable with machine learning, which is why I am going to let you in on the fact that data can be formatted one way or another, and it is up to you to understand how to get your datasets to work with your model.

Because the MNIST dataset is made for this purpose, it is already ready-to-use and little to no reshaping or reformatting has to go into this.

However, you might come across data you need to use for your model that is not that well formatted or ready for your machine learning model or scenario.

It is important to develop this skill, as in your machine learning career, you are going to have to deal with different types of data.

Now, let’s do the only reshaping we really need to do, reshaping the data to fit in out neural network input layer by converting it from a long matrix of pixel values to readable images. We can do this by adding the number of color channels as a dimension, and because the image is monochrome, we only need to add one as a dimension.

What is a Shape in Neural Networks?

A shape is the size of the linear algebra object you want to represent in code. I provide an extremely simple explanation of this here.

What is a Neural Network?

A neural network is a type of AI computers use to think and learn like a human. The type of neural network that we will be using today, sequential, models the human brain, consisting of layers of neurons that pass computed data to the next layer, which passes it’s computed data to the next layer, and so on, until it finally passes through the output layer, which will narrow the possible results down to however many output classes (desired amount of possible outcomes) you want. This whole layer cycle begins at the input layer, which will take the shape and pass it through to the rest of the layers.
Python
```
train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols, 1)
test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols, 1)
# Adding print statements to see the new shapes.
print((train_images.shape, test_images.shape))
```
Expected Output
```
((60000, 28, 28, 1), (10000, 28, 28, 1))
```
Now, we define the input shape, to be used when we define settings for the model.

What is an Input Shape?

An input shape defines the only shape that the input layer is capable of taking into the neural network.

We will begin data cleaning now, or making the data easier to process by the model.

First, let’s plot the digit 5 as represented in the MNIST dataset:
Python
```
plot_image(train_images, 100, train_labels)
```
This should output the following plot:

Now, let’s see what the numbers representing pixel intensity look like inside the image:
Python
```
out = ""
for i in range(28):
  for j in range(28):
    f = int(train_images[100][i][j][0])
    s = "{:3d}".format(f)
    out += (str(s)+" ")
  print(out)
  out = ""
```
Expected Output (Lines Providing no Useful Data are Blurred)
```
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   2  18  46 136 136 244 255 241 103   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0  15  94 163 253 253 253 253 238 218 204  35   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0 131 253 253 253 253 237 200  57   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0 155 246 253 247 108  65  45   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0 207 253 253 230   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0 157 253 253 125   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0  89 253 250  57   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0  89 253 247   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0  89 253 247   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0  89 253 247   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0  21 231 249  34   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0 225 253 231 213 213 123  16   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0 172 253 253 253 253 253 190  63   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   2 116  72 124 209 253 253 141   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  25 219 253 206   3   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 104 246 253   5   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 213 253   5   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  26 226 253   5   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 132 253 209   3   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  78 253  86   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0 
```
In order to help us visualize the data to another degree, let’s run the function below to show what the minimum and maximum values of the data are (the largest and smallest value in the data):
Python
```
show_min_max(train_images, 100)
```
Expected Output
```
0 255
```
Now we can start the actual data cleaning. As you saw above, the data in the image is represented as an integer between zero and 255. While the network could learn on this data, let’s make it easier for the network by representing these values as a floating point number between zero and one. This keeps the numbers small for the neural network.

Sign up for our newsletter!

Name

Email

By submitting your information, you’re giving us permission to email you. You may unsubscribe at any time.

Δ

First thing’s first, let’s convert the data to a floating-point number:
Python
```
train_images = train_images.astype('float32')
test_images = test_images.astype('float32')
```
Now that the data can be stored as a floating point number, we need to normalize the data all the way down to 0 to 1, not 0 to 255. We can achieve this by using some division:
Python
```
train_images /= 255 
test_images /=255
```
Now we can see if any changes were made to the image:
Python
```
plot_image(train_images, 100, train_labels)
```
The code above should output:

As you could see, no changes were made to the image. Now we will run the code below to check if the data was actually normalized:
Python
```
out = ""
for i in range(28):
  for j in range(28):
    f = (train_images[100][i][j][0])
    s = "{:0.1f}".format(f)
    out += (str(s)+" ")
  print(out)
  out = ""
```
Expected Output (Lines Providing no Useful Data are Blurred)
```
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.2 0.5 0.5 1.0 1.0 0.9 0.4 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.4 0.6 1.0 1.0 1.0 1.0 0.9 0.9 0.8 0.1 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 1.0 1.0 1.0 1.0 0.9 0.8 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 1.0 1.0 1.0 0.4 0.3 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 1.0 1.0 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.6 1.0 1.0 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 1.0 1.0 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.9 1.0 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.9 1.0 0.9 0.8 0.8 0.5 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 1.0 1.0 1.0 1.0 1.0 0.7 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.3 0.5 0.8 1.0 1.0 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.9 1.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.4 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.8 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.9 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 1.0 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 1.0 0.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
```
As you can see, the image is not affected, but the data is easier for the neural network to deal with.

If we don’t want to have to stifle through all those numbers but still check to see if we have cleaned the data correctly, let’s look at the minimum and maximum values of the data:
Python
```
print("The min and max are: ")
show_min_max(train_images, 100)
```
Expected Output (Lines Providing no Useful Data are Blurred)
```
The min and max are: 
0.0 1.0
```
We could start building the model now, but there is a problem we need to address. MNIST’s labels are simply the digits 1 to 9 because, well, the entire dataset is just handwritten digits 1 to 9. However, due to the nature of neural networks, they inherently believe that the data is ordered (i.e. 1 is more similar to 2 than 7, when in reality 7 looks more like the number 1, but they do this because from a mathematical perspective 1 is more similar to 2), which is wrong. To do this, convert the data to a categorical format, one that Keras won’t think is ordered, making it view each number independently:
Python
```
train_labels = keras.utils.to_categorical(train_labels, num_classes) 
test_labels= keras.utils.to_categorical(test_labels, num_classes)
```
This is also called One-Hot Encoding.

Now, we can finally start building our model.

Training done on datasets are called epochs. Each epoch is one complete pass over the entire dataset. Generally speaking, most epochs yeild more accurate results, but take a longer time to train. Finding the balance between reasonable time and good results is important when developing an AI model.

For now, we are just going to be training the model with ten epochs, but this number can be adjusted as you wish.
Python
```
epochs = 10
```
In this tutorial, we will be making a sequential model. In the future, you may need to make other types of models.

Defining our model:
Python
```
model = Sequential()
```
Now, we need to add the first layer (also called the input layer, as it takes input):
Python
```
model.add(Flatten(input_shape= (28,28,1)))
```
That layer is a flatten layer. It will convert the data into a long string of numbers, but in a way that the neural network can understand. We prepared the data for this earlier. Because it does not know what shape the data is stored as, we have to specify it in the input_shape parameter.

Now, we can add the layers needed.

We will add a Dense layer below, which will perform predictions on the data. We can configure a lot here, and in the future as a machine learning engineer, you will need to learn what the proper configurations for your scenario are. For now, we are going to use the activation function ReLU and put 16 neurons in this layer.

What is ReLU?

ReLU is an activation function that stands for Rectified Linear Unit. It uses the property of nonlinearity to properly rectify data sent through it. For example, if a negative number is passed through it, it will return 0.
Python
```
model.add(Dense(units=16, activation='relu'))
```
Finally, we will add the output layer. It’s job, as implied in the name, is to shrink the amount of possible outputs down to the number of output classes specified. Each output from this layer represents the AI’s guess on how likely one of its guesses is to be correct (in computer vision terms, this is known as the confidence).

We will make sure that the neural network shrinks this down to ten output classes (as the possible outputs are the digits zero to nine) by putting ten neurons into it (as you probably guessed, one neuron will output its guess on how likely it is that it’s correct), and by using the Softmax activation function to do so.

What is Softmax?

Softmax is an activation function that distributes the outputs such that they all sum to one. We are using it as the activation function for the final layer because our neural network is outputting something that could be interpreted as probability distribution.
Python
```
model.add(Dense(units=10, activation='softmax'))
```
Now, we can see an overview of what our model looks like:
Python
```
model.summary()
```
Expected Output (Lines Providing no Useful Data are Blurred)
```
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 flatten (Flatten)           (None, 784)               0         
                                                                 
 dense (Dense)               (None, 16)                12560     
                                                                 
 dense_1 (Dense)             (None, 10)                170       
                                                                 
=================================================================
Total params: 12,730
Trainable params: 12,730
Non-trainable params: 0
_________________________________________________________________
```
As you saw above, our model is sequential, has three layers that reshape the data, and already has 12,730 parameters to train. This means that the network is going to change 12,730 numbers in a single epoch. This should be enough to correctly identify a hand-drawn number.

Now, we have to compile the network and provide data to TensorFlow such that it compiles in the way that we want it to.
What do All the Arguments Mean?
The Optimizer is an algorithm that, as you probably guessed from the name, optimizes some value. Optimizing a value can mean either making it as big as possible or as small as possible. In a neural network, we want to optimize the loss (or how many times the neural network got the data wrong) by making it as small as possible. The optimizer is the function that does all this math behind the scenes. There are many functions for this, each with their own strengths or weaknesses. We will use Adam, a popular one for image recognition as it is fast and lightweight.

The Loss is the difference between a model’s prediction and the actual label. There are many ways to calculate this, which is why it is important to choose the right one. The loss function you need varies based on the how your neural network’s output should look like. For now, we should just use Categorical Cross Entropy.

The Metrics. For convenience purposes and to better visualize the data, TensorFlow allows the developer to choose which additional metrics it should show to supplement the metrics already shown during training. Accuracy, or what percent of input images the model guessed correctly, is one metric that can be visualized during training. It is similar to loss, but is calculated in a separate way, so accuracy and loss won’t necessarily add up to 100% or be direct inverts of each other.
Python
```
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
```
Once our model is compiled, we can fit the model to the training data that we prepared. We will use the actual training data to train the model in a way that lets it recognize numbers.

The train_images is the dataset that will be the inputs given to the model, while the train_labels will be like the answer to the questions, helping us keep track of if the network’s guess was correct or not. The epochs will be the amount of epochs it needs to run. This will be set to the variable we defined earlier.
Python
```
model.fit(train_images, train_labels, epochs=epochs, shuffle=True)
```
Expected Output (may vary)
```
Epoch 1/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.4289 - accuracy: 0.8818
Epoch 2/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2530 - accuracy: 0.9291
Epoch 3/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2187 - accuracy: 0.9387
Epoch 4/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1968 - accuracy: 0.9440
Epoch 5/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1815 - accuracy: 0.9491
Epoch 6/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1687 - accuracy: 0.9514
Epoch 7/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1605 - accuracy: 0.9539
Epoch 8/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1524 - accuracy: 0.9560
Epoch 9/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1459 - accuracy: 0.9574
Epoch 10/10
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1402 - accuracy: 0.9590
```
You can notice how, as the epochs progress, the loss goes down and the accuracy goes up. This is what we want!

However, knowing the labels to all the data basically makes those metrics useless – after all, you are just giving the model an answer – so we need to evaluate the model to see how well it could really do. We can achieve this by evaluating the model on test data – data the model has never seen before.

The <model>.evaluate function takes the testing data, as well as the trained model, and evaluates the model, producing a set of metrics (also called scores) that show how well the model really did on unforeseen data.

Although the function is taking the test labels, the function never shows this data to the neural network, only using it to grade the neural network on how well it did.
Python
```
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
```
Expected Output (may vary)
```
313/313 - 0s - loss: 0.1657 - accuracy: 0.9528 - 347ms/epoch - 1ms/step
```
As you saw above, both the loss and accuracy seem to be pretty low. This is because both the loss and accuracy are stored as precents in the form of decimals. This means that, for the output above, the loss is 16.57% and the accuracy is 95.28%. That is pretty good.

Using Our Model

First download this image to the same folder as the Python file, and name it test.jpg.

Now, run the code below to predict our image using <model>.predict:
Python
```
path = "test.jpg"

img = tf.keras.preprocessing.image.load_img(path, target_size=(28,28), color_mode = "grayscale")
x = tf.keras.preprocessing.image.img_to_array(img)
true_label = 3
p_arr = predict_image(model, x)
plot_value_array(p_arr, true_label, 1)
```
Expected Output (may vary)
```
Predicted Label: 2
...
```
It probably got the answer wrong. This is because it’s used to inverted images, meaning light handwriting on dark paper. To do this, we simply need to invert the image colors:
Python
```
x_inv = 255-x
```
And now we can run the prediction again:
Python
```
arr = predict_image(model, x_inv)
plot_value_array(arr, 3, 1)
```
Expected Output (may vary)
```
Predicted Label: 3
...
```
It probably got the answer correct. You have successfully built a neural network!

Exporting The Model

To do this, simply run the code below (which saves it to a file called my_model.h5:
Python
```
model.save('my_model.h5')
```
Now if you ever want to refer to it again in another file, simply load in the sequential model:
Python
```
model = keras.models.load_model("my_model.h5", compile=False)
```
Flaws in Our Code

There are flaws in out model. Firstly, if you tried evaluating it on multiple images, you may have noticed that it was not accurate. This is because if we want it to recognize an image, we have to optimize it for that image.

Because all of the training images were white on black, it has to do a lot of guessing when it gets confused on an image that is black on white.

We can fix this with convolutional neural networks.

It recognizes the small parts and details of an image, will be much more accurate, and will be better with more general data.

Follow along for the next part, where I teach you how to optimize this with convolutional neural networks.
Wednesday, July 19, 2023, 11:13 AM (GMT+0000)
How to Listen on ATC Conversations Using a SDR
Did you know that ATC conversations and conversations between planes are freely available, with no encryption? It is legal to listen in on ATC conversations, and in this guide I will tell you how if you have some free time.

What You Need

RTL-SDR Stick and Antenna (x1)

This is the antenna and radio processor we will be using to get a signal from an air traffic control tower.

Buy Now (~$27)

SDRSharp by Airspy

This is the program that we will be using to listen to these conversations and to tune the antenna.

Download (Free)

Initial Setup

If it is your first time using SDR# (SDRSharp), then you must install SDR#, then install the drivers. The below guide will show you how to do so.

First, install SDR# and let the installation wizard guide you through the process.

Then, open the newly added program Zadig and you should see a screen like the one below.
- A: This is where you choose the interface you want to install drivers for
- B: This is where you check if a driver was installed
- C: This is where you can install the drivers
Follow the steps below:
- First, use dropdown A to select an interface. The interface must start with Bulk-in, Interface. If you have multiple bulk-in interfaces, repeat these steps for every one
- Next, make sure textbox B tells you that there is no driver installed
- Finally, click Install WCID Driver (button C)
Opening SDR#

Once all the drivers are installed, you may close out of Zadig and open SDR#. You should see a screen like the one below.
- A: This is the frequency selector. This is where you can choose which frequency your antenna is supposed to be tuned to. Right now it is tuned to 120 MHz, but in the next section you will learn to find the frequency of your ATC tower
- B: This is where you can choose your radio settings. For this tutorial, keep the default settings but change the radio mode to AM
- C: This is where you choose the source of the radio stream. Right now you want it set to RTL-SDR USB
- D: This is where you can visualize the radio waves. You can click anywhere on this to set the frequency to the location of the waves to which you clicked. You can drag the lighter waves to set the bandwidth. You want to make sure that the bandwidth is not too big otherwise you will get interference, but not too small so you only get part of the wave. I have set my bandwidth to 7.557 kHz
Reading Aerospace Vector Maps

Using a site, like SkyVector, you can find your airport and look at the frequency under it. Tune to that frequency. For place value context, think of the second segment of numbers as MHz SkyVector shows frequencies in megahertz.

Some airports, like the ones marked with a star, do not have full-time ATC, meaning that planes have to talk directly to each other.

Tune to this frequency on SDR#.

Listening to these frequencies

Look for any spikes in these frequencies. Ste the frequency to the frequency of these spikes (you can do this easily by clicking on these spikes) Adjust the bandwidth to these spikes, hovering over the top-right Zoom button and using the slider below it to zoom into the waves. Click on the top-left gear icon and adjust the setting to match the ones below:

Now, turn the volume up and listen. If you do not hear talking, experiment with the bandwidth or choose another frequency. A good frequency should be like the one below:

Done!

And that is the end of the project! Pretty easy, right? There are some caveats, though. You will only get the best signal when you live no further than 50 kilometers away from an airport with a full-time ATC, and the radio tends to disconnect a lot if not screwed in fully. Either way, it is still a super cool project, and is definitely worth trying out if you are interested in this kind of thing. Frequencies might not be exact, so experiment a little!
Tuesday, June 13, 2023, 12:38 AM (GMT+0000)