Some things you might have heard being thrown around in the cryptography space are encoding methods. Such encoding methods are normally used to turn binary data into text, typically so it can be used to transmit data over communication methods that do not allow raw binary data to be transmitted.
How does binary turn into Base64?
First, let’s say that we want to encode an image. We start by reading all the bits in the image, which, for this example, will result in a binary stream of 010010101101001000111111. It is important to note that most images are a lot larger than this and will contain hundreds of thousands of bits.
The encoder will first break the binary stream up into their own chunks, each containing six bits. We can do this to our example and get 010010 101101 001000 111111.
Then, we turn our binary chunks into decimals.
To do this, working right to left, we multiply each bit by 2 to the power of p, where p starts from zero and increments by one for every bit in the chunk.
For example, we can calculate 010010 by performing the expression (0 * 20) + (1 * 21) + (0 * 22) + (0 * 23) + (1 * 24) + (0 * 25). We can strip away any parts that multiply by zero, since those will always be zero. The simplified equation is (1 * 21) + (1 * 24) = 2 + 16 = 18. Therefore, 010010 is equivalent to the decimal 18.
Turning every single six-bit binary chunk into a decimal using the process shown above constitutes the numbers (in left-to-right order) 18, 45, 8, and 63.
After that, we can simply cross reference the numbers with whatever alphabet we should use. For example, the six-bit chunk 010010 turns into the decimal 18, which corresponds to S on the Base64 standard alphabet. Therefore, 010010 101101 001000 111111 turns into StI/.
How does binary turn into Base64?: fixing the padding
If the binary string is a multiple of 3, everything fits perfectly. However, if it is not, then we need to add padding to the encoded string until the string’s bit count becomes a multiple of 3.
Let’s take 010010 101101 001000 111111 01 as an example. It is 26 bits long and all full six-bit chunks encode to StI/. However, this is leaving out the two bits at the end.
To pad this, we will keep adding zeroes to the end of the string until it can become a full chunk. Thus, the final chunk, 01, becomes 010000, which corresponds to Q in the Base64 alphabet. Then, we add one padding character (typically an =) for every two bits we had to add for the string to only be made of full chunks. We add two padding characters because four (the number of bits we had to add) divided by two is two.
Thus, you would think that the padded string would be StI/Q==, right? Well, no, because that extra character adds another six bits, meaning that there are still two bits at the end. We can only stay in such a loop two times, so we need to add more zeroes. That means that the final letter is 000000, which is A.
So, the final result is StI/QA==.
How does Base64 turn into binary?
Base64 decoding is just like base64 encoding, but in reverse. First, we turn every single non-padding character into its respective index on the alphabet, then we turn every single number into its binary equivalent.
For example, let’s take our string StI/QA==. We first take every single non-padding character and turn it into a string of numbers that represents their indexes on the alphabet.
This turns our Base64-encoded string into a string of numbers, specifically 18 45 37 62 16 0 = =, since we don’t turn the padding into a number
In an earlier post I made, I discussed how regular expressions could be used. Now, I will show you how to implement them in your own C++ program.
The regex Library
The regular expression library was added in C++ 11, so you should have support for it by now. We can start with some basic boilerplate, importing our regular expression library (along with some other libraries that will make our lives easier) and the standard IO library.
In C++, regular expressions must be compiled before they are used. When I say I am going to be passing a regex string or a regex to a function, I am actually going to be talking about this compiled expression. All regexes must be compiled before use.
It is actually very easy to compile regexes, and below, we are compiling regex <html>.+</html> and assigning the compiled expression to a variable called re.
Then, we will do the same thing I mentioned above but naming the expression reg. The reason I am doing this twice is because I want to show the two methods you can use, assigning by value or assigning using the regex class’s constructor.
Determining if a Regular Expression Matches an Entire String
The regex_match function will determine whether an entire string is matched by a certain regex. For example, if we pass the regex hi to it and match it with the string hi, the function will return true, as the regular expression provided matches the entire target string of hi.
However, if we kept the regex the same but changed the target string to shi, the function would return false because while shi contains hi, the regex hi does not match the entirety of shi.
Let’s use an example. I have given one below.
C++
#include<iostream>#include<string>#include<vector>#include<regex>usingnamespacestd;intmain(){ string reStr; cout<<"Enter a regular expression to use the regex_match function on:\n>"; cin>>reStr; string target; cout<<"Enter a target string to use the regex_match function on:\n>"; cin>>target; regex reCompiled =regex(reStr); // Compiling our regex // Actual matching processif(regex_match(target,reCompiled)){ cout<<"\nRegex Matched Entirely!\n";return0;}else{ cout<<"\nRegex Did Not Match Entirely!\n";}return-1;}
A Quick Note: Capturing Groups
Capturing groups in regexes are denoted by parenthesis and are often returned as lists. To make things simpler, let’s use the regex (sub)(mar)(ine). Here we can see that sub, mar, and ine each have their own capturing groups.
Now if we were to use this on the text submarinesubmarine, the regex would match on both submarines separately, so we would get two matches.
Let’s take a closer look at the matches.
These matches would end up having three submatches each due to these capturing groups. If we were to visualize this in hierarchy, we would get the following:
In C++, matching with capturing groups is represented as a list of matches containing lists of each capturing group for each match. For example, if we wanted to get match one, capturing group one, of a list of matches (you will learn about the smatches type in the next section), we would use the code below:
C++
string m1c1 = matches[0][0];
A Quick Note: The smatches Type
The smatches type is used for storing a list of strings as regex matches. It is sort of like a vector, but the shape is fixed to either vector<string> without capturing groups, or vector<vector<string>> with capturing groups.
Determining if a Regular Expression Matches any Substrings
Remember how above, I said that the regex_match function only tells you whether the entire string is matched by a regex? Well, if we want to include substrings, it can get a little more complicated (this is coming from someone with a Python background, where we are pampered with the re library).
For this part of the guide, we will be using the regex_search function, which will tell you if
The regex_search function typically takes three to four arguments. Let’s look at the first method of calling it.
For this method, the function takes three parameters and outputs a Boolean. The parameters are below.
Target (std::string) – This is the string you want to match the regex against
Match Results (std::smatch) – This is the variable of type smatch that will store match results. We will not be using it in this example
Regex (std::basic_regex) – This is the compiled regex that the target is being matched against
The function will return true if any substring of the target string matches the regex, and false otherwise.
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ string s="this variable is called s"; smatch m; regex e =regex("s");if(regex_search(s,m,e) /* Will return true */){ cout<<"Matched! (but not always the entire string)"<<endl;}return0;}
We can also call regex_search using another method, whose parameters are listed below. In this method, we are not only telling the user whether the program is
String Begin and String End – Tells the function to only search the substring in between the string beginning and string ending
Match Results – This is the smatch that will store match results
Regex – The compiled regex that will be used to match against the target string
The function will return true using the same conditions I stated in the previous method, but here what we care about is the fact that the match results are being stored.
The code below will print the first match of the regex, check if there are any matches other than the one it returned, and print the first capturing group. It will keep doing this until there are no other matches. I highly recommend you read the comments in the code below for a better understanding of what it’s doing.
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ string target ="submarine submarine submarine"; regex re =regex("(sub)(mar)(ine)"); smatch m;string::const_iterator searchFrom =string::const_iterator(target.cbegin()); // Begin iteratingwhile(regex_search(searchFrom,target.cend(),m,re)){ // We don't want to keep returning the same match every time, so the code below will exclude this match from the future iterations searchFrom =m.suffix().first; // It is important to know that m[0] would return the entire string ("submarine"), so m[1] will return the first capturing group ("sub") cout<<"We have got ourselves a match! \""<<m[1].str() /* First capturing group of match */<<"\"\n";}}
Regular Expression Find and Replace
The regex_replace function will find and replace all sequences that match the regex.
In the example below, we are telling it to replace all words (including the spaces around them) with “and”
We are also giving it three parameters.
Target – The text that will be replaced accordingly
Regex – The compiled regular expression that will be used on the target
Replace With – The text to replace the matches of the regex with against the target
C++
#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ regex re("([^ ]+)"); // Matches every word cout<<"ORIGINAL: this is text\n"; cout<<regex_replace("this is text",re,"and"); // prints "and and and"return0;}
You can also use formatters to incorporate exactly what was replaced using the table below.
Formatter
Example
Explanation
$number(where “number” is replaced by any positive number less than 100)
$2
Replaced with the match of the numberth capturing sequence that triggered the replace (starting from 1, such that $1 will get the first capturing group, not $0) at runtime
Example: Replacing regex matches of “(sub)(.+)” with “2nd CG: $2” using a target string of “submarine” will yield a result of “2nd CG: marine”
$&
$&
A copy of the entire original string, regardless of capturing groups.
Example: Replacing regex matches of “(sub)(.+)” with “String: $&” using the same target string above will result in “String: submarine”
$`
$`
Replaced with whatever came before the match at runtime
Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$`” will get replaced with “a “
$’
$’
Replaced with whatever came after the match at runtime
Example: When we have a regex of “sub” with target string “a submarine goes underwater”, “$’” will get replaced with “marine goes underwater”
$$
$$
I wouldn’t call it a formatter exactly; it’s more of an escape sequence. Used when you don’t want the compiler to mistake the literal character “$” with a formatter.
Used when you want to literally type “$” as the text to replace, type “$$”
For example, the code below will put the letters “t” and “e” in parenthesis.
C++
// regex_replace example#include<iostream>#include<string>#include<regex>usingnamespacestd;intmain(){ regex re("([te])"); // Matches either "t" or "e" cout<<"ORIGINAL: thetechmaker.com\n"; cout<<regex_replace("thetechmaker.com",re,"($&)"); // Prints "(t)h(e)(t)(e)chmak(e)r.com"return0;}
In this guide, I will show you how to use the BeautifulSoup library to make a simple program that notifies you when a product on an online site drops in price.
This library runs in the background, scraping static online e-commerce sites of your choice and notifying you when a product drops in price.
Prerequisites
This guide assumes that you have Python installed, pip added to your system’s PATH, along with a basic understanding of Python and HTML.
Installing Required Components
First, let’s install BeautifulSoup and Requests. The Requests library retrieves our data, but the BeautifulSoup library actually analyzes our data.
We can install those two required components by running the command below:
BAT (Batchfile)
pip install beautifulsoup4 requests
Note that depending on what your system’s setup is, you might need to use pip3 instead of pip.
Grabbing Our Sample: Price
In this step, we will be telling BeautifulSoup what exactly to scrape. In this case, it’s the price. But we need to tell BeautifulSoup where the price is on the website.
To do this, navigate to the product you want to scrape. For this guide, I will be scraping an AV channel receiver I found on Amazon.
Then, use your browser’s DevTools and navigate to the price. However, make sure that you have a very “unique” element selected. This is an element that shows the product’s price but is also very specifically identified within the HTML document. Ideally, choose an element with an id attribute, as there cannot be two elements with the same HTML ID. Try to get as much “uniqueness” as you can because this will make the parsing easier.
The elements I have selected above are not the most “unique” but are the closest we can get as they have lots of classes that I can safely assume not many other elements have all of.
We also want to ensure that our web scraper stays as consistent as possible with website changes.
If you also don’t have an element that is completely “unique”, then I suggest using the Console tab and JavaScript DOM to see how many other elements have those attributes.
Like, in this case, I am trying to see whether the element I selected is “unique” enough to be selected by its class.
In this case, there is only one other element that I need to worry about, which I think is good enough.
Basic Scraping: Setup
This section will detail the fundamentals of web scraping only. We will add more features as this guide goes on, building upon the code we will write now.
First, we need to import the libraries we will be using.
Python
import requests as rqfrom bs4 import BeautifulSoup
Then, we need to retrieve the content from our product. I will be using this AV receiver as an example.
If the content you want to scrape is locked behind a login screen, chances are you need to provide basic HTTP authentication to the site. Luckily, the Requests library has support for this. If you need authentication, add the auth parameter to the get method above, and make it a tuple that follows the format of ('username','password').
For example, if Amazon required us to use HTTP basic authentication, we would declare our request variable like the one below:
If that authentication type does not work, then the site may be using HTTP Digest authentication.
To authenticate with Digest, you will need to import HTTPDigestAuth from Request’s sub-library, auth. Then it’s as simple as passing that object into the auth parameter.
Python
from requests.auth import HTTPDigestAuthrequest = rq.get("https://www.amazon.com/Denon-AVR-X1700H-Channel-Receiver-Built/dp/B09HFN8T64/",auth=HTTPDigestAuth("replaceWithUsername","replaceWithPwd"))
If the content you want to scrape requires a login other than basic HTTP authentication or Digest authentication, consult this guide for other types of authentications.
Amazon does not require any authentication, so our code will work providing none.
Now, we need to create a BeautifulSoup object and pass in our website’s response to the object.
When you use the Requests library to print a response to the console, you generally will want to use request.text. However, since we don’t need to worry about decoding the response into printable text, it is considered better practice to return the raw bytes with request.content.
Basic Scraping: Searching Elements
Now we can get to the fun part! We will find the price element using our sample we got earlier.
I will cover two of the most common scenarios, one where you need to find the price based on its element’s ID – the simplest, or one where you need to find the price based on class names and sub-elements – a little more complicated but not too difficult, assuming you have a “unique” enough element.
If we wanted to refer to an element based on its ID with BeautifulSoup, you would use the find method. For example, if we wanted to store the element with the ID of pricevalue within a variable called priceElement, we would invoke find() with the argument of id set to the value "pricevalue".
Python
priceElement = parser.find(id="pricevalue")
We can even print our element to the console!
Python
print(priceElement.prettify())
Expected Output (may vary)
<divid="pricevalue"><p>$19.99</p></div>
The function prettify is used to reformat (“pretty-print”) the output. It is used when you want to be able to visualize the data, as it results in better-looking output to the console.
Now we get to the tougher part – making references to element(s) based on one or more class names. This is the method you will need to use for most major e-commerce sites like Amazon or Ebay.
This time, we will be using the find_all function. It is used only in situations where it is theoretically possible to get multiple outputs, like when we have multiple classes as the function gives the output as a list of strings, not a single string.
If you are not sure, know that you can use find_all even when the query you give it only returns one result, you will just get a one item list.
The code below will return any elements with the classes of priceToPayorbig-text.
The select function is just like that of the find function except instead of directly specifying attributes using its function parameters, you simply pass in a CSS selector and get a list of matching element(s) back.
The code above selects all elements with the class of both price-value and main-color. Although many use the find or find_all functions, I prefer select as I am already familiar with CSS selectors.
If, and this is not much of a good idea when finding elements, we would like to filter by element type, we will just call find_all with a single positional argument, the element’s type. So, parser.find_all("p") will return a list of every single paragraph (“p“) element.
An element type is one of the broadest filters you can pass into the find_all function, so this only becomes useful when you combine it with another narrower filter, such as an id or class.
Python
parser.find_all("h1",id="title")
That would return all h1 elements with an ID of title. But since each element needs to have its own unique ID, we can just use the find function. Let’s do something more realistic.
Python
parser.find_all("h1",class_="bigText")
This code would return all h1 elements that had a class of bigText.
Below are a few reviews of what we know so far and some other, rarer methods of element finding.
Python
"""Never recommended, but returns a list of ALL the elements that have type 'p'"""typeMatch = parser.find_all("p")"""Finds element with the ID of 'priceValue' using a CSS selector"""idSelMatch = parser.select("#priceValue")"""Finds element with the ID of 'priceValue', except with the BeautifulSoup-native find function and not with a CSS selector"""idMatch = parser.find(id="priceValue")# Same as above"""Extremely rare, but returns a list of elements containing an ID of 'priceValue' OR 'price'"""orIdMatch = parser.find_all(id=["priceValue","price"])"""Returns a list of elements that have the class 'price' OR 'dollarsToPay'. I do not know of a CSS selector that does the same"""orClassMatch = parser.find_all(class_=['price','dollarsToPay'])"""Returns a list of elements that have the class 'price' AND 'dollarsToPay'. I do not know of a find_all argument that does the same"""andClassMatch = parser.select(".priceValue.dollarsToPay")"""Returns the element that has a class of 'v' INSIDE the element of class 't'. This can also be done with ID attributes, but this function only works when the first function is .find(...) or when you are grabbing an element by index after calling .find_all(...). Because .find(...) only returns one element, it will only be returning the first instance of that class name. The code below return the same thing, however 'inMatch3' returns a list"""inMatch = parser.find(class_="t").find(class_="v")# Most basic way to do itinMatch2 = parser.find_all(class_="t")[0].find_all(class_="v")[0]# Because .find_all(...) works on the final element, the '[0]' is unnecessary, we just do it so we don't get a one-element listinMatch3 = parser.find_all(class_="t")[0].find_all(class_="v")# Returns a one-element list
Now that we know how to search elements, we can finally implement this in our price drop notifier!
Let’s see if our request is successful. We will be printing out the entire file to check.
Python
print(parser.find("html").prettify())
And we are not.
Hmmm, so we have to bypass Amazon’s CAPTCHA somehow, so let’s try adding headers that mimic a normal browser!
I will be adding headers to rq.get(). Make sure to replace my AV channel receiver link with the product you want to scrape.
Nope. Still nothing. Well, time for plan B, ditching requests completely and using selenium.
Sign up for our newsletter!
Basic Scraping: Implementation of Selenium
Firstly, it is important to know that Selenium has its own methods for finding elements in a HTML document, but for the sake of this guide, we will just be passing the source code of the website to our parser.
Think of Selenium as a browser running in the background with some selection abilities. Instead of sending the requests to the website by crafting our own headers, we can use Selenium to spin up an invisible browser that crafts the headers for us. We should no longer get a CAPTCHA screen because Amazon shouldn’t be suspicious that a robot is browsing the page – we are technically using a legitimate browser, but with parsing capabilities.
Installation of Selenium can be done with the command below. We will also be installing win10toast so you get a proper toast notification whenever a price drop is detected.
BAT (Batchfile)
pip install seleniumpip install win10toast
If you are looking for how you can uninstall Requests because you don’t need it anymore, think twice because Selenium depends on Requests anyways.
Now, clear your entire Python file because we are going to need to do a short and quick rewrite of our code to use Selenium.
Like always, we will start by importing the required modules. Make sure you replace chrome with the name of a browser you have installed on your system, preferably the most resource efficient one.
Python
from selenium import webdriverfrom bs4 import BeautifulSoupfrom selenium.webdriver.chrome.options import Options # Imports the module we will use to change the settings for our browserimport time # This is what we will use to set delays so we don't use too many system resourcesfrom win10toast import ToastNotifier # This is what we will use to notify if a price drop occurs.notifier =ToastNotifier()# Assign our notifier class to a variable
Then, we will need to set some preferences for the browser we are about to start. Let’s start by declaring an Options class and using it to make the browser invisible or run it in “headless” mode. While the arguments below are for specific browsers, I would just execute them all because I have not tested each argument individually.
Python
browserOptions =Options()browserOptions.headless=True# Makes Firefox run headlessbrowserOptions.add_argument("--headless=new")# Makes newer versions of Chrome run headlessbrowserOptions.add_argument("--headless")# Makes older versions of Chrome run headlessbrowserOptions.add_argument("--log-level=3")# Only log fatal errors
Now, we will initiate the browser in the background. Again, make sure you replace Chrome with whichever browser you want to use for this project.
Then, we can use what we already know about BeautifulSoup to grab the price of our element. Remember to replace the code below with one tailored to your sample.
Next, let’s strip the $ symbol from the price and convert it into a floating-point decimal.
Python
price =float(price.strip("$"))
Then, we can set a variable to compare with the current price.
Python
previousPrice = price
Now, we loop infinitely to see whether the price changed.
Python
whileTrue:
Insert a new line and then indent the code we will write from this point forward.
Now, every two minutes (120 seconds), we refresh the page and compare the price we just got to our previous price.
Python (place each line indented inside while loop)
browser.refresh()# Refreshes the browser# Now that we may have a new price, we have to redfine our parser and price variables to adapt to that new page codeparser =BeautifulSoup(browser.page_source,"html.parser")price = parser.select(".a-price.aok-align-center.reinventPricePriceToPayMargin.priceToPay")[0].find_all(class_="a-offscreen")[0].textprice =float(price.strip("$"))# Next, we compare the two prices. If we find one, we alert the user and update our price threshold. We will also be looking for price increases.if(price<previousPrice):print(f"Price DECREASED from ${previousPrice} to ${price}!") notifier.show_toast("Price Drop!",f"The price decreased from ${previousPrice} to ${price}!")elif(price>previousPrice):print(f"Price INCREASED from ${previousPrice} to ${price}!") notifier.show_toast(":(",f"The price increased from ${previousPrice} to ${price} :(")# Now, we can tell the user we refreshedprint(f"Refreshed! Previous Price: ${previousPrice}, and new price ${price}")previousPrice = price# And then we wait for two minutestime.sleep(120)
And just like that, you are finished! I hoped this project was useful to you!
Regular expressions (abbreviated as a “regex”) are useful tools that help easily find and match text in strings and files. They sort of function like the typical find-in-document feature you are probably used to, only more advanced. However, they can definitely be too advanced for an average person to comprehend. In this guide, I will easily explain this widely used and useful feature so you can search files like a pro.
Something Worth Noting
It is worth noting that regexes are very useful and have lots of features to utilize, but can also be very non-standardized. There are always features in other regex parsing engines that are not supported by others. The features I will be displaying in this guide are the most widely supported ones, but there are always going to be features that I won’t teach you how to use here either because they are not known or because they are defunct and not considered good practice.
Basic Matching
Regexes can match specific strings, no complex syntax needed. For example, to match the text foo in notnotnotfoonotnot, your regex would simply be foo.
This works with multiple instances too; the regex foo would match both instances of foo in notnotnotfoonotnotfoo.
Something else that you should know about basic matching is that there will be two seperate matches if the same matching string is used multiple times (together or not).
This means that the regex foo would match all three foo instances in notnotfoofoonotfoo (notice how there are separate matches for the two instances of foo, even though said instances are together).
Match Any Character
The period selects any character. Note that each character is matched separately.
For example, the regex . in the text foo will match f, o, and the second o.
Notice how in the image above, even special characters and numbers are matched.
Letter Range Match
Let’s say in the text abcdefghijklmnopqrstuvwxyz, we want to match just e, f, g, h, and i. To do this, we can select the letter range that we want (in this case, e through i) using the regex [e-i].
Note that these letter ranges do not have to be subsequent; we can have the regex [a-c] in the text azaczbca match a, a, c, b, c, and a.
Number Range Match
Let’s say in the text 019201836y7 you only want to select the numbers 0, 1, 2, the second 0, and the second 1. In other words, we want to select any number from zero to two, including themselves. We can do this using the number range match, and in this case, we can use the regex [0-2].
These letter ranges also don’t have to be consecutive.
Character Set
Now let’s say you want to match specific characters that can’t be matched with a set. For this, we will use a character set. For example, we can match car and bar in the text car,bar using a character set. Because only the beginning letters of each word change, we can use a character set to match the c and b, and then use basic matching to match the ars. To do this, we can put in the characters we want to put in the set in square brackets.
Ignored Character Set
How about if we wanted to match every character except for c and b. To do this we can use an ignored character set to get the job done. If we wanted to match only tar in the text car,bar,tar, we can use an ignored character set to ignore c and b, then use basic matching as all three words have different initials but the same of everything else. To do this, we can put the characters we want to ignore in square brackets with the character ^ coming before them.
The Asterisk
The asterisk (*), when placed outside a character indicates that said character should be matched even if it does not occur at all or occurs at least once side by side. For example, the regex a* will match at least zero of a, so aaaa will be matched in the text aaaab. This is different from basic matching, as the regex a will count every a as a separate match, while a* will keep all the as in one match as many times as needed to complete the match.
Another examples is that the asterisk is like saying the character that comes before it is optional or can occur multiple times in a row. The regex ab*c in the text abc,abbc,ac will match abc, abbc, and ac.
The Question Mark
The question mark is just like the asterisk in the sense that the character that comes before it is deemed optional, but different in the sense that it will not match characters that occur multiple times in a row. For example, the regex ab?c will match abc and ac, but not abbc.
The Plus Symbol
The plus symbol is also just like the asterisk in the sense that the character that comes before it can occur multiple times in a row, but different in the sense that it does not indicate a character is optional. If we go back to our abc example, we can see that the regex ab+c in abc,abbc,ac will match abc and abbc, but not ac.
Conclusion
And that’s it – you’re done! There are so many other regex sequences you can use, and they just won’t fit into a beginner’s guide. As you saw above, regular expressions are very useful tools, and there are many ways to apply this to your needs.
In this guide, I will be showing you how to install Arch Linux from scratch. You can do this on a physical machine, but I will be doing it on a virtual machine.
What is Arch Linux?
Arch Linux is a minimal Linux distro that is meant for power users and advanced programmers. It does not come with a built-in installer, so we will have to install it manually.
Downloading the ISO
First, head to the Arch Linux downloads page, scroll down until you find the download mirrors, and then choose a link to download, preferably from your country so you get the fastest download speed.
VM Configuration
If you are installing Arch Linux on physical hardware and you are not using a VM to install Arch, skip this section.
If you are on VirtualBox, there should be an Arch Linux preset. If you are on VMware, select Other Linux 5.x Kernel (64-bit).
Giving your VM 8 GB of RAM is a lot more than needed, but if you are going to be using Arch for power-intensive tasks and don’t mind the VM taking up all your host’s RAM, go for whatever fits for you as long as it meets the system requirements of 512 MBminimum, but 2 GB recommended for streamlined daily use.
Now, give your VM any amount of storage you feel fitting, but make sure it meets the system requirements of 1 GBminimum, but 20 GBrecommended.
If you want things to go a little faster on your Arch Linux VM, giving it two processors is recommended. One processor should be enough, though.
If you plan on installing a desktop environment, enable 3D acceleration and give a reasonable amount of VRAM to the guest OS.
And lastly, make sure the CD drive is set to read as the ISO you just downloaded.
Making a Bootable USB
If you are using a VM, skip this section.
Use a tool like Rufus to flash the Arch Linux ISO to a flash drive. Then plug it into the system you want to install Arch Linux to. This guide does not cover dual-booting and assumes you do not have an existing OS installed on your system.
Boot Priority Configuration
Make sure the hard disk is the first on the boot priority list. This is not required; it just makes the final step of the installation a lot quicker.
You can do this in the VM’s settings, the UEFI firmware settings, or the BIOS interface.
Starting the System
If you are on a VM, start it up. If you are on hardware, plug in the thumb drive that you flashed the ISO to and start the machine.
You should see an Arch Linux splash screen. Whichever entry comes first in the list is likely the one that boots to Arch (the name of the entry changes, but typically goes along the lines of “Boot Arch Linux (x86_64)”). Select Boot Arch Linux, or the entry that does so, and click Enter.
After running some tests, you should be dropped into a root Linux shell. Do not remove your thumb drive or installation media, as we have not installed Arch yet and need to do so using the thumb drive.
Accessing the Internet
Let’s check if we have connection to the internet by running the command below (“google.com” can be replaced with a website of your choice):
Bash
pinggoogle.com
You should see packets return. If you do, this means you have online access, which is a necessity for this installation. You can hit Control + C to stop pinging the website.
If you are on a Virtual Machine with no internet, try making sure that you have host internet access, and then try enabling the network adapter in the VMware or VirtualBox settings.
If you are on a physical machine without internet, try using an Ethernet cable, but below you will know how to use iwctl to connect to a Wi-Fi network.
Using iwctl to Connect to a Wi-Fi Network
This part of the guide should only be followed if you do not have Internet access and want to use a wireless internet connection instead of a wired one.
Run the command below to open the iNet Wireless Daemon CTL.
Bash
iwctl
Expected Output
[iwd]#
Now, you can list the wireless adapters using the command below.
Bash
devicelist
You should see a device called wlan0. It is best to go forward using that one, but you can select another wireless adapter if you know what you are doing. If you do not have that device, then you do not have a wireless adapter plugged into the computer, the device is connected but under a different name (which is unlikely), or the computer does not recognize it.
Use the command below to list all the wireless networks found using that device. You can replace wlan0 with the adapter you chose earlier.
Bash
stationwlan0get-networks
You should see a list of networks. Take note of the network’s SSID you want to connect to. Then, you can run the command below to connect to the wireless network, replacing “WirelessNet” with the SSID of your wireless network and wlan0 with the wireless adapter you want to connect to the network using.
Bash
stationwlan0connectWirelessNet
After typing in the Wi-Fi password (if needed), you may now connect to the network. You can test your connection by using ping google.com and waiting for packets to return. If none return, then you might have done something incorrectly when setting up the network.
Setting NTP Time
Now that we have our network up and running, we can enable network time using the command below:
Bash
timedatectlset-ntptrue
Partitioning
Now comes the tricky part. We have to partition our drives manually, so make sure to follow my steps carefully.
Run the below command to get a summary of all the partitions on your drive.
Bash
lsblk
Make sure you choose the right disk to partition, as choosing the wrong one will destroy all of your data. Run the below command to set up partitioning, replacing /dev/sda with the name of the disk you want to format and install Arch Linux on.
As for the label type, it depends what your needs are. If you are installing to a new physical system with a disk size larger than 2TB, select gpt and hit Enter. If you either don’t have a physical system to install Arch to or are installing Arch to a disk smaller than 2TB, use dos. Now, at the menu with drives listed, select the free space and click New. When asked for your partition size, enter the amount needed for your bootloader. If you will be using the GRUB bootloader, enter 128M and hit Enter. If not, specify the amount needed for your bootloader.
Now, select the newly created partition and hit B to make the selected drive bootable.
Select the free space, and click New. The size should automatically default to the remaining storage on your drive. Make the partition fill up the rest of the drive, and click Enter to create the partition. You should not make this partition bootable.
Many people prefer creating home and swap partitions but these are mostly considered redundant nowdays.
Select Write and click Enter, then select Quit and click Enter.
There is only one problem left to solve now – the drives are not in the format of ext4. To solve this, run the commands below, replacing /dev/sda1 and /dev/sda2 with the names your newly created boot and OS partitions.
Bash
mkfs.ext4/dev/sda1mkfs.ext4/dev/sda2
Mounting our Partitions
Now that we are done with arguably the hardest part of Arch installation, we need to mount our drives, which is where the preparation ends and the actual installation starts.
To begin, lets mount our root partition (the partition that is not bootable, /dev/sda2 in my case) to a mount point (this can be anything you want, but traditionally this has always been /mnt. I will be using /mnt, as I do not see any reason to stray from tradition in my case). We can do this using the command below.
Bash
mount/dev/sda2/mnt
In our mount point, let’s create a folder called boot to mount our other drive to.
Bash
mkdir/mnt/boot
Now, let’s mount our boot partition (the one that we flagged bootable earlier, /dev/sda1 in my case) to the folder we just created.
Bash
mount/dev/sda1/mnt/boot
To see if we did everything correctly, we can run the command below.
Bash
lsblk
In the output, you should be able to see partitions under the drive /dev/sda and their respective mount points.
Installing Arch Using pacstrap
Now, we can begin installing system files to Arch.
We can use pacstrap to install Arch Linux and some other packages we want pacstrap to bundle in with our Arch installation. Replace /mnt with the mount point you mounted your root drive to, and vim with some other text editor that you prefer and some other pre-installed packages you want on Arch.
Bash
pacstrap/mntbasebase-devellinuxlinux-firmwarevim
Below ae explanations of some of the packages:
Base: This package contains basic tools that you would want no matter which Linux distribution you are installing.
Base-Devel: This contains developer tools such as compilers, which are needed for some Linux components.
Linux: This is the core Linux kernel that runs everything behind the scenes.
Linux-Firmware: This contains firmware that makes Arch compatible with common hardware
Vim: This can be replaced with any text editor. There is some text editing we are going to have to do, so we need a text editor.
Once pacstrap exits, we can generate an fstab file that lists all the drives Linux could try booting from.
Generating our FSTab File
This is extremely easy to do. Run the command below, replacing /mnt with the mount point you specified earlier.
Bash
genfstab-U/mnt>>/mnt/etc/fstab
What the command should do is write a FSTab file to the drives. The -U flag makes the file reference drives by UUID instead of drive name. This is good because drive names may change in the future, which could mess up your boot configuration if you don’t reference drives by UUID, as UUID never changes.
Jumping to our Arch Installation
It is finally time to change from our installation medium to our disk. Do not remove your installation medium until the end of this guide. You might need it if something breaks.
Bash
arch-chroot/mnt/bin/bash
After this command, do not reboot yet. We still have to install our boot manager.
Installing Basic Components
Now that we are in Arch, we have access to the Pacman package manager. We can use it to install basic components like a network manager for accessing the internet, and a boot manager so we can boot into the system. I will be installing GRUB as a boot manager, but you can install something else.
Use the command below to install these components.
Bash
pacman-Snetworkmanagergrub
Configuring the Network Manager to Run on Startup
If we want internet at boot, we are going to have to enable NetworkManager’s system service. we can do this using the command below.
Bash
systemctlenableNetworkManager
Configuring the GRUB Boot Manager
Now, we have to configure what our system boots using. When we ran the Pacman command, we downloaded GRUB, but we did not install or configure it. Let’s install GRUB using the command below, replacing /dev/sda with whatever drive you are using to install Arch. We are not going to be using /dev/sda1 or /dev/sda2 because it is critical that you install it to the drive, not the drive partition.
Bash
grub-install/dev/sda
Now, we can make a GRUB configuration file using the command below.
Bash
grub-mkconfig-o/boot/grub/grub.cfg
Take a look at the output and make sure that it says both an initrd image and a Linux image were found. If it does not find these images, the most likely cause is incorrectly installing the Linux kernel using pacstrap.
Setting a Password
Now, run the command below to create a password for your root user.
Bash
passwd
Configuring Locales and Timezones
Use the command below to enter the locale configuration file, replacing vim with whatever text editor you installed earlier.
Bash
vim/etc/locale.gen
Now, use the arrow keys or the scroll wheel to scroll down to the locale you want to use. I will be using United States English, so I will scroll down to en_US and uncomment (remove the # before) both the UTF and ISO formats. If you are using Vim, you might have to hit the I key on your keyboard before you can actually type anything.
Write the file and exit your text editor. To write and exit Vim, we can hit the Escape key on our keyboard and type :wq.
Now that we have our locales configured, we have to apply the changes by generating the locales. We can do this using the command below.
Bash
locale-gen
Now, we also have to create a file called locale.conf to define our language in. Use the command below, once again replacing vim with your desired text editor.
Bash
vim/etc/locale.conf
In the file, type LANG=en-US.UTF-8, once again replacing en-US with whatever locale you are using. Exit Vim.
Now that we have the timezones prepared, we have to link them to make our system clock show the right timezone. Type ln -sf /usr/share/zoneinfo and click Tab. This will list all the broadest timezones. Keep making the directories more specific, hitting Tab to see the available options every time, and after you are done typing a city, hit Space and type /etc/localtime.
Setting Our Hostname
Now, we can set our hostname. This is the name that the Arch machine will use to communicate with other devices over your network. By default, your hostname is archiso. If you are happy with that and don’t want to change it, you can skip this section.
Use your prefered text editor to create /etc/hostname. Do not include a file extension. Type whatever you want your system hostname to be, and exit.
Unmounting Our Disk
Now, we can exit our chroot jail by using the command below.
Bash
exit
Now would be a good time to check and make sure your hard drive is first boot priority. Make sure that when you return, you are in your installation medium and not in the actual Arch installation.
Unmount our Arch installation with the command below, replacing /mnt with the mount point you specified earlier.
Bash
umount-R/mnt
Now, we can boot into our new installation of Arch Linux using the command below. Once you have booted in, you may remove the installation medium.
This guide is a part two to a previous guide I made, called The Simple Guide to AI and Machine Learning With Python. This guide is simply how you can improve accuracy to the model you made in that guide, meaning that I’m going to assume you have already completed the previous guide before going on to follow this guide.
In the previous guide, we learned how you can use dense neural networks to make a program that recognizes handwriting. Well, that neural network was not exactly very accurate, as it had a tendency to get numbers wrong unless it was specifically modified for those numbers. As you probably know by now, you would probably want the neural network to recognize any number you give it without having to optimize the network for every single number that comes to it.
Convolutional neural networks were made to solve this problem. Rather than training off of the overall image, convolutional neural networks recognize tiny features in the image and learns those. For example, rather than focusing on the entire image of a hand-drawn three, the network will learn that a three has two curves that are stacked vertically, which will help it recognize any other threes in the future, no matter how it was drawn or whether the neural network was optimized for the number three.
Step One: Initial Setup
For this step, we can just use the code that we used in the previous tutorial to prepare the MNIST dataset.
Python
import tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras.datasets import mnistfrom tensorflow.keras import backend as Kimport numpy as npimport matplotlib.pyplotas plt%matplotlib inline# helper functionsdefshow_min_max(array,i): random_image = array[i]print("min and max value in image: ", random_image.min(), random_image.max())defplot_image(array,i,labels): plt.imshow(np.squeeze(array[i])) plt.title(" Digit "+str(labels[i])) plt.xticks([]) plt.yticks([]) plt.show()defpredict_image(model,x): x = x.astype('float32') x = x /255.0 x = np.expand_dims(x,axis=0) image_predict = model.predict(x,verbose=0)print("Predicted Label: ", np.argmax(image_predict)) plt.imshow(np.squeeze(x)) plt.xticks([]) plt.yticks([]) plt.show()return image_predictimg_rows, img_cols =28,28num_classes =10(train_images, train_labels),(test_images, test_labels)= mnist.load_data()(train_images_backup, train_labels_backup),(test_images_backup, test_labels_backup)= mnist.load_data()print(train_images.shape)print(test_images.shape)train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols,1)test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols,1)input_shape =(img_rows, img_cols,1)train_images = train_images.astype('float32')test_images = test_images.astype('float32')train_images /=255test_images /=255train_labels = keras.utils.to_categorical(train_labels, num_classes)test_labels = keras.utils.to_categorical(test_labels, num_classes)print(train_images[1232].shape)
Expected Output
(60000, 28, 28)(10000, 28, 28)(28, 28, 1)
Now that we have already put in the initial setup of our code, we can jump straight to creating our network.
Creating Our Network
Similar to what we did with the densely connected network, we are still going to have epochs, or the amount of times the network goes through the entire set over again.
With that explanation out of the way, we can define our model.
Python
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropoutepochs =10model =Sequential()
Now, let’s start adding the layers of our neural network.
Explaining Convolutional Layers
With our previous network, we added three dense (fully connected) layers. With our new network that uses convolutional neural networks, the layers work differently.
Convolutional layers consist of groups of neurons called filters that move across the image and activate based on the pixels they read. Those groups will then learn how to recognize features in the data.
It is possible to adjust amount and size for filters in your neural network, which we will change to our liking. Bigger filters can observe larger parts of the image at once, while smaller filters gather finer details about the image. A higher amount of filters means that the neural network can recognize a wider range of image features.
There are many advantages of having layers and filters work this way. For one thing, smaller filters can be more computationally efficient by only examining a small part of the image at once. Furthermore, as filters are moved across the entire image, the neural network will not be affected by feature displacement (occurs when a feature is common to two images, but in different spots of an image). Just like reality, filters focus on a small area of the image, so they are not distracted by the other parts of an image.
We will be using multiple convolutional layers to complete our new-and-improved handwriting recognition software.
Implementing Convolutional Layers
When we use Keras, we can easily take advantage of its functionality to easily create convolutional layers that we will then use in our model. We will use the Conv2D function to create the first layer of out neural network.
In the case below, we will have 32 filters, a kernel size of (3,3), an input shape – which we saved to the input_shape variable when we ran the setup code at the beginning – of (28,28,1), and an activation function of ReLU. I go more in-depth into what ReLU is in my previous guide.
The Conv2D function creates 2D convolutional layers, meaning that they scan across flat data, like images.
Explaining Pooling Layers
When you use convolutional layers, things can get quite computationally intensive, which is where pooling layers come in. Increasing the number of neurons will increase the number of computation time required. Pooling layers are essentially filters that move in specified strides across the image, simplifying each of the filters’ contents into a single value. This, based on the size and stride of the filter, shrinks the output image.
For this scenario, we will have a 2×2 filter with a size of 2. This halves the image’s row and column count, simplifying the data without too much loss of specificity.
Python
model.add(MaxPooling2D(pool_size=(2,2)))
Most networks have at least one set of alternating convolutional and pooling layers.
More Convolutional Layers
Convolutional layers are designed to examine the low-level features of an image. If we add more, we may be able to start working with higher-level features.
We define the layer the same way we defined the previous one, but now we have 64 filters, not 32. We also do not specify the input shape, as it is inferred from the previous layer.
Dropout layers are layers that take a percentage of all input neurons and deactivate them randomly. This forces other neurons to adapt to the task. When larger and more complicated networks lack a dropout layer, the network risks being too dependent on a single set of neurons rather than all neurons learning. This is called Overfitting and can change your network output for the worse.
Below, we will have our dropout layer take 30%, or 0.3 neurons to deactivate randomly.
Python
model.add(Dropout(rate=0.3))
Dense and Flatten Layers
After all the convolutional and pooling layers, we will need a layer to help make our final decision. This will be a regular, fully connected dense layer. Before we connect this layer, we will need to flatten the image’s filters.
We can start by flattening the image using the Keras Flatten layer.
Python
model.add(Flatten())
Now, we can add a dense layer with ReLU activation and 32 neurons.
Python
model.add(Dense(units=32,activation='relu'))
Sign up for our newsletter!
Output Layers
Similar to the fully connected neural network we made in the previous guide, we will need a layer to shrink the previous dense layer down to just the number of classes. Also similar to before, the final output is decided by using the class with the highest weight.
Below, we will add a dense layer to be our output layer. The number of neurons should be 10 because there are ten possible output classes, and the activation should use Softmax.
Python
model.add(Dense(units=10,activation='softmax'))
Model Summary
Now, we can print out our model summary:
Python
model.summary()
Expected Output (Lines Providing no Useful Data are Blurred)
Now we will compile the network. The loss and metric will be the same as the ones that we use in the previous guide, Categorical Cross Entropy and accuracy respectively. However, we will use RMSProp (Root Mean Squared Propagation) as our training algorithm. RMSProp is one of many training algorithms that Keras can use to teach the network how to actually improve, optimizing the loss to make it as small as possible. We will achieve this using RMSProp.
The fit function is the one that actually does the training.
Now we can look at the parameters of the training function.
train_images and train_labels state the data that this neural network model will be trained on. The images are the pieces of data given to the network, and the network tries to find out the appropriate label
batch_size allows us to put the network’s data into batches. We can always change it later, but for now we have set it to 64
epochs defines the number of epochs (times the network reiterates on the training data) the network should use
validation_data defines the data the model is testing itself on
We have turned shuffle on so Keras shuffles the training data after every epoch and isn’t relying on the order of the data to train on
Now, we have to test the model on data it hasn’t seen yet. To do this, we will use the evaluate function. Loss and accuracy are percentages returned in decimal format.
In this guide, you will learn how to create an AI that recognizes handwriting with Python using Dense neural networks and the MNIST dataset. This guide will use TensorFlow to train your AI, and basic knowledge of linear algebra used in AI is strongly recommended. You can refer to this guide to understand the linear algebra used in AI. In the next part, we upgrade the neural network’s accuracy using convolutional neural networks.
Prerequisites
To do this, you will first need to install Python and add Pip to the .bashrc file for Linux or the Environment Variables in Windows or Mac. Then, run the command below to install the required libraries:
Note: If installing TensorFlow does not work, you can run pip install tensorflow. This will function like normal, but it will not be able to utilize your GPU.
Writing The Code
In a new Python file, we will first import the dataset and import the libraries needed:
Python
import tensorflow as tffrom tensorflow import kerasfrom tensorflow.keras.datasets import mnistfrom tensorflow.keras import backend as Kimport numpy as npimport matplotlib.pyplotas pltfrom tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten
We then define some functions that will help us visualize the data better later on in the code. I will not go over how they work, but they are not a necessity, just there to help us visualize the data better:
In the MNIST Data set (the dataset that we will be using), there are 60,000 training images and 10,000 test images. Each image is 28 x 28 pixels. There are 10 possible outputs (or to be more technical, output classes), and there is one color channel, meaning that each image is stored as a 28 x 28 grid of numbers between 0 and 255. It also means that each image is monochrome.
We can use this data to set some variables:
Python
img_rows =28# Rows in each imageimg_cols =28# Columns in each imagenum_classes =10# Output Classes
Now, we will load the train images and labels and load in another set of images and labels used for evaluating the model’s performance after we train it (these are called test images/labels).
What Are Images and Labels?
These can also be data and labels. The data is the context that the computer is given, while the labels are the correct answer to predicting based on data. Most of the time, the model tries predicting labels based on the data it is given.
The next step is not required, and we don’t make use of it throughout the code, however it is recommended, especially if you are using a Python notebook.
The next step is to create a duplicate, untouched version of the train and test data as a backup:
Now, we test to see if we loaded the data correctly:
Python
print((train_images.shape, test_images.shape))
Expected Output
((60000, 28, 28), (10000, 28, 28))
Why Are They Those Shapes?
The images are 28×28, so that explains the last two dimensions in the shape. Because the data is stored as a long matrix of pixel values (this is not readable to our neural network, by the way; we will fix this later), we do not need to add any more dimensions. If you remember what I said earlier, you will know that there are 60000 training images and 10000 testing images, so that explains the first dimension in the tensor.
The whole purpose of this tutorial is to get you comfortable with machine learning, which is why I am going to let you in on the fact that data can be formatted one way or another, and it is up to you to understand how to get your datasets to work with your model.
Because the MNIST dataset is made for this purpose, it is already ready-to-use and little to no reshaping or reformatting has to go into this.
However, you might come across data you need to use for your model that is not that well formatted or ready for your machine learning model or scenario.
It is important to develop this skill, as in your machine learning career, you are going to have to deal with different types of data.
Now, let’s do the only reshaping we really need to do, reshaping the data to fit in out neural network input layer by converting it from a long matrix of pixel values to readable images. We can do this by adding the number of color channels as a dimension, and because the image is monochrome, we only need to add one as a dimension.
What is a Shape in Neural Networks?
A shape is the size of the linear algebra object you want to represent in code. I provide an extremely simple explanation of this here.
What is a Neural Network?
A neural network is a type of AI computers use to think and learn like a human. The type of neural network that we will be using today, sequential, models the human brain, consisting of layers of neurons that pass computed data to the next layer, which passes it’s computed data to the next layer, and so on, until it finally passes through the output layer, which will narrow the possible results down tohowever many output classes (desired amount of possible outcomes) you want. This whole layer cycle begins at the input layer, which will take the shape and pass it through to the rest of the layers.
Python
train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols,1)test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols,1)# Adding print statements to see the new shapes.print((train_images.shape, test_images.shape))
Expected Output
((60000, 28, 28, 1), (10000, 28, 28, 1))
Now, we define the input shape, to be used when we define settings for the model.
What is an Input Shape?
An input shape defines the only shape that the input layer is capable of taking into the neural network.
We will begin data cleaning now, or making the data easier to process by the model.
First, let’s plot the digit 5 as represented in the MNIST dataset:
Python
plot_image(train_images,100, train_labels)
This should output the following plot:
Now, let’s see what the numbers representing pixel intensity look like inside the image:
Python
out =""for i inrange(28):for j inrange(28): f =int(train_images[100][i][j][0]) s ="{:3d}".format(f) out +=(str(s)+"")print(out) out =""
Expected Output (Lines Providing no Useful Data are Blurred)
In order to help us visualize the data to another degree, let’s run the function below to show what the minimum and maximum values of the data are (the largest and smallest value in the data):
Python
show_min_max(train_images,100)
Expected Output
0 255
Now we can start the actual data cleaning. As you saw above, the data in the image is represented as an integer between zero and 255. While the network could learn on this data, let’s make it easier for the network by representing these values as a floating point number between zero and one. This keeps the numbers small for the neural network.
Sign up for our newsletter!
First thing’s first, let’s convert the data to a floating-point number:
Now that the data can be stored as a floating point number, we need to normalize the data all the way down to 0 to 1, not 0 to 255. We can achieve this by using some division:
Python
train_images /=255test_images /=255
Now we can see if any changes were made to the image:
Python
plot_image(train_images,100, train_labels)
The code above should output:
As you could see, no changes were made to the image. Now we will run the code below to check if the data was actually normalized:
Python
out =""for i inrange(28):for j inrange(28): f =(train_images[100][i][j][0]) s ="{:0.1f}".format(f) out +=(str(s)+"")print(out) out =""
Expected Output (Lines Providing no Useful Data are Blurred)
As you can see, the image is not affected, but the data is easier for the neural network to deal with.
If we don’t want to have to stifle through all those numbers but still check to see if we have cleaned the data correctly, let’s look at the minimum and maximum values of the data:
Python
print("The min and max are: ")show_min_max(train_images,100)
Expected Output (Lines Providing no Useful Data are Blurred)
The min and max are: 0.0 1.0
We could start building the model now, but there is a problem we need to address. MNIST’s labels are simply the digits 1 to 9 because, well, the entire dataset is just handwritten digits 1 to 9. However, due to the nature of neural networks, they inherently believe that the data is ordered (i.e. 1 is more similar to 2 than 7, when in reality 7 looks more like the number 1, but they do this because from a mathematical perspective 1 is more similar to 2), which is wrong. To do this, convert the data to a categorical format, one that Keras won’t think is ordered, making it view each number independently:
Training done on datasets are called epochs. Each epoch is one complete pass over the entire dataset. Generally speaking, most epochs yeild more accurate results, but take a longer time to train. Finding the balance between reasonable time and good results is important when developing an AI model.
For now, we are just going to be training the model with ten epochs, but this number can be adjusted as you wish.
Python
epochs =10
In this tutorial, we will be making a sequential model. In the future, you may need to make other types of models.
Defining our model:
Python
model =Sequential()
Now, we need to add the first layer (also called the input layer, as it takes input):
Python
model.add(Flatten(input_shape=(28,28,1)))
That layer is a flatten layer. It will convert the data into a long string of numbers, but in a way that the neural network can understand. We prepared the data for this earlier. Because it does not know what shape the data is stored as, we have to specify it in the input_shape parameter.
Now, we can add the layers needed.
We will add a Dense layer below, which will perform predictions on the data. We can configure a lot here, and in the future as a machine learning engineer, you will need to learn what the proper configurations for your scenario are. For now, we are going to use the activation function ReLU and put 16 neurons in this layer.
What is ReLU?
ReLU is an activation function that stands for Rectified Linear Unit. It uses the property of nonlinearity to properly rectify data sent through it. For example, if a negative number is passed through it, it will return 0.
Python
model.add(Dense(units=16,activation='relu'))
Finally, we will add the output layer. It’s job, as implied in the name, is to shrink the amount of possible outputs down to the number of output classes specified. Each output from this layer represents the AI’s guess on how likely one of its guesses is to be correct (in computer vision terms, this is known as the confidence).
We will make sure that the neural network shrinks this down to ten output classes (as the possible outputs are the digits zero to nine) by putting ten neurons into it (as you probably guessed, one neuron will output its guess on how likely it is that it’s correct), and by using the Softmax activation function to do so.
What is Softmax?
Softmax is an activation function that distributes the outputs such that they all sum to one. We are using it as the activation function for the final layer because our neural network is outputting something that could be interpreted as probability distribution.
Python
model.add(Dense(units=10,activation='softmax'))
Now, we can see an overview of what our model looks like:
Python
model.summary()
Expected Output (Lines Providing no Useful Data are Blurred)
As you saw above, our model is sequential, has three layers that reshape the data, and already has 12,730 parameters to train. This means that the network is going to change 12,730 numbers in a single epoch. This should be enough to correctly identify a hand-drawn number.
Now, we have to compile the network and provide data to TensorFlow such that it compiles in the way that we want it to.
What do All the Arguments Mean?
The Optimizer is an algorithm that, as you probably guessed from the name, optimizes some value. Optimizing a value can mean either making it as big as possible or as small as possible. In a neural network, we want to optimize the loss (or how many times the neural network got the data wrong) by making it as small as possible. The optimizer is the function that does all this math behind the scenes. There are many functions for this, each with their own strengths or weaknesses. We will use Adam, a popular one for image recognition as it is fast and lightweight.
The Loss is the difference between a model’s prediction and the actual label. There are many ways to calculate this, which is why it is important to choose the right one. The loss function you need varies based on the how your neural network’s output should look like. For now, we should just use Categorical Cross Entropy.
The Metrics. For convenience purposes and to better visualize the data, TensorFlow allows the developer to choose which additional metrics it should show to supplement the metrics already shown during training. Accuracy, or what percent of input images the model guessed correctly, is one metric that can be visualized during training. It is similar to loss, but is calculated in a separate way, so accuracy and loss won’t necessarily add up to 100% or be direct inverts of each other.
Once our model is compiled, we can fit the model to the training data that we prepared. We will use the actual training data to train the model in a way that lets it recognize numbers.
The train_images is the dataset that will be the inputs given to the model, while the train_labels will be like the answer to the questions, helping us keep track of if the network’s guess was correct or not. The epochs will be the amount of epochs it needs to run. This will be set to the variable we defined earlier.
You can notice how, as the epochs progress, the loss goes down and the accuracy goes up. This is what we want!
However, knowing the labels to all the data basically makes those metrics useless – after all, you are just giving the model an answer – so we need to evaluate the model to see how well it could really do. We can achieve this by evaluating the model on test data – data the model has never seen before.
The <model>.evaluate function takes the testing data, as well as the trained model, and evaluates the model, producing a set of metrics (also called scores) that show how well the model really did on unforeseen data.
Although the function is taking the test labels, the function never shows this data to the neural network, only using it to grade the neural network on how well it did.
As you saw above, both the loss and accuracy seem to be pretty low. This is because both the loss and accuracy are stored as precents in the form of decimals. This means that, for the output above, the loss is 16.57% and the accuracy is 95.28%. That is pretty good.
Using Our Model
First download this imageto the same folder as the Python file, and name it test.jpg.
Now, run the code below to predict our image using <model>.predict:
It probably got the answer wrong. This is because it’s used to inverted images, meaning light handwriting on dark paper. To do this, we simply need to invert the image colors:
It probably got the answer correct. You have successfully built a neural network!
Exporting The Model
To do this, simply run the code below (which saves it to a file called my_model.h5:
Python
model.save('my_model.h5')
Now if you ever want to refer to it again in another file, simply load in the sequential model:
Python
model = keras.models.load_model("my_model.h5",compile=False)
Flaws in Our Code
There are flaws in out model. Firstly, if you tried evaluating it on multiple images, you may have noticed that it was not accurate. This is because if we want it to recognize an image, we have to optimize it for that image.
Because all of the training images were white on black, it has to do a lot of guessing when it gets confused on an image that is black on white.
We can fix this with convolutional neural networks.
It recognizes the small parts and details of an image, will be much more accurate, and will be better with more general data.
Follow along for the next part, where I teach you how to optimize this with convolutional neural networks.
Did you know that ATC conversations and conversations between planes are freely available, with no encryption? It is legal to listen in on ATC conversations, and in this guide I will tell you how if you have some free time.
What You Need
RTL-SDR Stick and Antenna (x1)
This is the antenna and radio processor we will be using to get a signal from an air traffic control tower.
If it is your first time using SDR# (SDRSharp), then you must install SDR#, then install the drivers. The below guide will show you how to do so.
First, install SDR# and let the installation wizard guide you through the process.
Then, open the newly added program Zadig and you should see a screen like the one below.
A: This is where you choose the interface you want to install drivers for
B: This is where you check if a driver was installed
C: This is where you can install the drivers
Follow the steps below:
First, use dropdown A to select an interface. The interface must start with Bulk-in, Interface. If you have multiple bulk-in interfaces, repeat these steps for every one
Next, make sure textbox B tells you that there is no driver installed
Finally, click Install WCID Driver (button C)
Opening SDR#
Once all the drivers are installed, you may close out of Zadig and open SDR#. You should see a screen like the one below.
A: This is the frequency selector. This is where you can choose which frequency your antenna is supposed to be tuned to. Right now it is tuned to 120 MHz, but in the next section you will learn to find the frequency of your ATC tower
B: This is where you can choose your radio settings. For this tutorial, keep the default settings but change the radio mode to AM
C: This is where you choose the source of the radio stream. Right now you want it set to RTL-SDR USB
D: This is where you can visualize the radio waves. You can click anywhere on this to set the frequency to the location of the waves to which you clicked. You can drag the lighter waves to set the bandwidth. You want to make sure that the bandwidth is not too big otherwise you will get interference, but not too small so you only get part of the wave. I have set my bandwidth to 7.557 kHz
Reading Aerospace Vector Maps
Using a site, like SkyVector, you can find your airport and look at the frequency under it. Tune to that frequency. For place value context, think of the second segment of numbers as MHz SkyVector shows frequencies in megahertz.
Some airports, like the ones marked with a star, do not have full-time ATC, meaning that planes have to talk directly to each other.
Tune to this frequency on SDR#.
Listening to these frequencies
Look for any spikes in these frequencies. Ste the frequency to the frequency of these spikes (you can do this easily by clicking on these spikes) Adjust the bandwidth to these spikes, hovering over the top-right Zoom button and using the slider below it to zoom into the waves. Click on the top-left gear icon and adjust the setting to match the ones below:
Now, turn the volume up and listen. If you do not hear talking, experiment with the bandwidth or choose another frequency. A good frequency should be like the one below:
Done!
And that is the end of the project! Pretty easy, right? There are some caveats, though. You will only get the best signal when you live no further than 50 kilometers away from an airport with a full-time ATC, and the radio tends to disconnect a lot if not screwed in fully. Either way, it is still a super cool project, and is definitely worth trying out if you are interested in this kind of thing. Frequencies might not be exact, so experiment a little!
Before we start this project, you need to know a little bit about GPS. Satellites will send signals on their distance from the module. Once four or more satellites are connected and giving out this data, the receiver can then use the data to figure out the exact location of the user.
The receiver will then present this data in the form of NMEA sentences. This is a standard communication developed by the National Marine Electronics Association. We will be using TinyGPS++ to parse these.
Using the GPS with Arduino
For this part, you don’t need the LCD. This will show you how to log the GPS output to the serial monitor. We will then parse this data using TinyGPS++.
Preparations
First, open Arduino IDE and you will be greeted with a blank sketch.
At the top toolbar, click Ctrl + Shift + I to bring up the library manager and type “TinyGPSPlus” and install the top result.
Sign up for our newsletter!
Code
Now that we are all prepared, lets start writing the code. First, we include the library that helps us communicate with the GPS.
#include <SoftwareSerial.h>
Next, we include the library that parses NMEA sentences.
#include <TinyGPSPlus.h>
Now, declare the communication between the Arduino and the GPS and then the parser.
SoftwareSerial ss(4,3);
TinyGPSPlus gps;
After that, we go inside the void setup() function and we initiate the communication between the computer and the Arduino and the GPS and Arduino.
Serial.begin(9600);
ss.begin(9600);
Next, we go into void loop() and specify that whatever is below this line of code should only happen when the Arduino receives a signal.
while (ss.available() > 0)
Then, we encode the data in a format that the GPS can then parse.
gps.encode(ss.read());
Then, we create an if block so the serial monitor only displays our data when the data the GPS is outputting is valid.
if (gps.location.isUpdated()) {
}
Now, inside the if block, we can access all the data and print it to the serial monitor.
Serial.print("Latitude= ");
Serial.print(gps.location.lat(), 6); //6 for 6 decimal places
Serial.print("Longitude= ");
Serial.print(gps.location.lng(), 6); //6 for 6 decimal places
Your full code should look like this:
#include <SoftwareSerial.h>
#include <TinyGPSPlus.h>
SoftwareSerial ss(4,3);
TinyGPSPlus gps;
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
ss.begin(9600);
}
void loop() {
// put your main code here, to run repeatedly:
while (ss.available() > 0)
gps.encode(ss.read());
if (gps.location.isUpdated()) {
Serial.print("Latitude= ");
Serial.print(gps.location.lat(), 6); //6 for 6 decimal places
Serial.print("Longitude= ");
Serial.print(gps.location.lng(), 6); //6 for 6 decimal places
}
}
Wiring
The wiring is shown below:
GPS RX > Digital 4 on Arduino
GPS TX > Digital 3 on Arduino
GPS VCC > Power 3.3V on Arduino
GPS GND > Power GND on Arduino
Uploading
Now, with the Arduino IDE opened and the code ready, press Ctrl + U on your keyboard. The code will write and start outputting to the serial monitor, which you can access by pressing Ctrl + Shift + M on your keyboard or by going to the top toolbar and clicking Tools > Serial Monitor. The GPS will take a couple minutes to get its location. You may want to stick the antenna outside for this, as it will take a long time to get its location indoors.
Soon, you will be able to view the data coming in.
If you had ever wanted to get the MacOS experience without owning a Mac, then you may want to use a virtual machine to get the experience, and this guide will teach you how to do just that by installing MacOS Ventura on a virtual machine.
NOTE This file is not from me and I simply found it on the internet. As of right now, VirusTotal did not get any positives on this file and it appears to work fine. This may change in the future, so beware.
NOTE You may be able to find a cracked version or use a pirated license key, but that is illegal and not recommended.
Step 1: Patching VMware Pro
Download the unlocker script from GitHub, and path VMware Workstation Pro. Make sure VMware Pro is patched before you do this, otherwise you might not install the patch correctly.
After running the patch, you should see MacOS as an option for your VM.
Step 2: Creating the VM
In the VM settings, create a new VM with the version of MacOS 12 and supply your ISO. Give the VM at least 16 gigabytes of RAM, and at least 80 gigabytes of hard drive storage. Give the VM a name, and click “Finish”
Step 3: First Boot and Setup
Click on Power on this virtual machine. Once the operating system has loaded, select your language and double-click on Disk Utility
Look for VMware Virtual SATA Hard Drive Media. Select it, then click Erase.
A menu will pop up. Change the name to whatever you want, and set the scheme to whatever you want. I set it to MacOS Extended (Journaled), but the most common format is GUID Partition Map. I would still recommend MacOS Extended, though. After you type your name and your scheme, click Erase.
The virtual disk will now begin erasing and formatting.
Once the virtual disk is done formatting, close out of the window and click Install MacOS 13 Beta.
The installer will then continue. The install should take around 15-20 minutes.
Once the VM finishes installing, wait for the machine to fully boot. Once it is bootes, shut down the VM and go to the VM’s settings.
Click on CD/DVD (SATA) and make sure that Connect at power on is disabled.
Power on the VM and continue with setup.
NOTE There have been problems with people’s screen going black and then rebooting after setting up network options. Make sure you do not set up Wi-Fi during setup, and set it up when you are in the desktop environment.
VMware Tools (Darwin)
Click on VM > Install VMware Tools… and continue with the setup on MacOS
Limitations
The following are limitations that you might want to consider before trying this project:
