While playing with ChatGPT, I found myself wanting to see its output.
ChatGPT can create HTML code, and it functions within a web browser. Since the browser is able to display HTML, is it possible to use it to display ChatGPT’s output?
Indeed, with just a bit of glue code we can see and interact with the output!
Here is some JS that, when evaluated, will render ChatGPT’s output (and yeah, ChatGPT helped me write this). Note I only tested this in Firefox.
function replaceLastCodeBlock() {
var codeBlocks = document.getElementsByTagName("code");
var lastCodeBlock = codeBlocks[codeBlocks.length - 1];
if (!lastCodeBlock) {
return;
}
var htmlContent = lastCodeBlock.innerText;
var fragment = document.createRange().createContextualFragment(htmlContent);
lastCodeBlock.parentNode.replaceChild(fragment, lastCodeBlock);
var elements = document.getElementsByClassName("bg-black");
for (var i = 0; i < elements.length; i++) {
elements[i].classList.remove("bg-black");
}
}
replaceLastCodeBlock();
Disclaimer: It’s probably not super secure to haphazardly evaluate code produced by a machine learning model; use at your own risk.
Let’s see some examples…
Here’s a simple showcase. The script above makes the browser render ChatGPT’s output.
CSS and JavaScript are evaluated.
ChatGPT can produce SVG code, which also can be rendered.
beautiful
Obligatory recursive ChatGPT in an iframe.
You can make the script feed input back into ChatGPT.
And data can be fetch
ed from the internet.
Combining data retrieval and feedback, you can jury-rig more advanced prompting with context. This is pretty brittle, and notice it erroneously outputs its knowledge cutoff year.
Animations can be rendered
A fun Game
ChatGPT’s ability to generate correct code is impressive in its own right. Being able to easily see and interact with the evalated artifacts of that textual output makes the tool more fun. Hopefully this little demo is thought-provoking, enjoy!
]]>Like many software engineers, each week I receive multiple emails from recruiters.
I’m grateful to work in a field with such opportunities, and I know that receiving a lot offers to interview is a good problem to have. But, practically, most of the time I’m not looking for a new job, and so handling all these emails is a recurring administrative task.
Here’s an example thread that I neglected to respond to:
I do try to respond to all recruiter emails with a short message that pretty much always follows a format like:
Hi <recruiter name>,
Thanks for reaching out! I’m not interested at this time, but I’ll keep <your company> in mind.
- Matt
There are a few reasons that I respond to these emails (rather than merely ignore them):
I use some rough email filtering rules to funnel recruiter emails to an email folder. Then, when I have time, I go through the list of unreads and send my little response.
It would be ideal if I could automate sending these responses. Assuming I get four such emails per week and that it takes two minutes to read and respond to each one, automating this would save me about seven hours of administrative work per year.
A trivial approach would be to send a canned response. But a touch of personalization would aid in my goal of maintaining a good relationship with the recruiter.
Extracting the name of the recruiter and their company from the email using a rule-based approach / trying to parse the text would be really tricky and error prone. Luckily, OpenAI’s GPT-3 language model is quite good at processing this email.
Using the GPT-3 API, we can provide the recruiter’s email along with an example, and extract the required information. It can even format the output as JSON.
Here’s an example from the OpenAI Playground.
With the recruiter’s name and company in hand, responding is just a matter of interpolating those variables into the body of my standard response template:
response = f"""\
Hi {recruiter_name or ""},
Thanks for reaching out! I'm not interested in new opportunities at this time, but I'll keep {recruiter_company or "your company"} in mind for the future.
Thanks again,
{SIGNATURE}
"""
IMAP and SMTP are used to interface with the mailbox. The rest of the code can be found in this repo.
This solution worked well for the handful of emails I tried it on. I’m planning to run this on a cron to save myself some time and automatically maintain recruiter relationships.
]]>This post describes a solution I hacked together to cast video without Chrome (and without an Android emulator). It relies on VLC to do the actual casting. VLC is capable of (a) opening video from a network stream, and (b) using a Chromecast as the video renderer.
The script just copies the URL of the current Firefox tab and then invokes the VLC command line interface, passing in the video URL. I invoke the script with Alfred when FF is browsing to a YouTube video (haven’t tested with other streams).
use scripting additions
use framework "Foundation"
on get_url_ff()
tell application "Firefox" to activate
set thePasteboard to current application's NSPasteboard's generalPasteboard()
set theCount to thePasteboard's changeCount()
tell application "System Events"
keystroke "l" using {command down}
delay 0.2
keystroke "c" using {command down}
end tell
-- hacky heuristic to help ensure the URL is copied
repeat 20 times
if thePasteboard's changeCount() is not theCount then exit repeat
delay 0.1
end repeat
set the_url to the clipboard
return the_url as text
end get_url_ff
on chromecast_tab_ff()
set the_url to get_url_ff()
-- how to get chromecast IP https://old.reddit.com/r/Chromecast/comments/8nu0d7/how_to_find_chromecasts_ip/dzyenff/
do shell script "/Applications/VLC.app/Contents/MacOS/VLC " & quoted form of the_url & " --sout \"#chromecast\" --sout-chromecast-ip=192.168.0.198 --demux-filter=demux_chromecast"
end chromecast_tab_ff
chromecast_tab_ff()
In terms of time, it took me 4.5 years to complete the program. I was working full time during this period, so I only took one course per semeseter (except for Fall 2020 when I doubled up). I also didn’t take any classes during the summer semesters. Fitting school around my work schedule was doable. My normal routine was school work three weeknights and one weekend day. I probably averaged ~10 hours per week on coursework and studying, though the workload varied depending on the class. I was able to earn a 4.0, and I never felt like I was doing an unreasonable amount of work. The course workloads aggregated on OMSCentral seem relatively accurate, but personally I think I spent less time than what’s listed there.
The program did sometimes put a strain on my social life, but from spring 2020 onward we were under COVID-19 restrictions anyway. It was often a drag to finish an entire day of work, only to then have to study for a test or implement a programming assignment. I know people complete their degree while caring for dependents, and it’s hard to imagine how they make it work. Now that I’m done, I’m glad to have some more free time back in my life.
In direct financial terms, the entire degree costed almost $8k, some of which was covered by my employer. Whether the degree has paid (or will pay) for itself is unclear. I’m not intending, in my next job search, to target only jobs for which an M.S. is required. I believe I’m a stronger engineer for having completed the program, but any future success in my career probably won’t be directly attributable. My OMSCS specialization was Machine Learning, but I don’t intend to pivot my career to an ML focus. “Artificial Intelligence” has an aura of extreme hype, and I think my ML specialization has helped me to regard AI with a more informed and critical perspective.
I think the experience served the goals I’d set for it. Namely, it gave me a structured way to learn more about sub-fields of CS that I wasn’t normally exposed to in my daily work as a web software engineer. Probably most importantly, it strengthened my learning ability, which is of course hugely applicable. Fighting impostor syndrome is an ongoing battle, but I think being able to understand the course material also helped in that regard.
My advice to OMSCSers…
The rest of this post is a list of the courses I took and some brief notes about each one…
I took this course first because it was supposedly really challenging and really good. I wanted to see what I was getting myself into. It was indeed pretty hard! The programming assignments were in C and C++, with which I was rusty. I’d also been out of college for five years. The course material was about distributing massive workloads on supercomputers using tools like MPI. The recorded lectures were engaging and entertaining and the lab assignments were nontrivial. In retrospect, it was a rewarding and fun class despite being challenging. I’m glad I took it first. I probably spent the most time on this course.
This class was a broad introduction to statistical methods like regression, Q-Learning, and KNN, and to financial concepts like market mechanics, valuing companies, and technical analysis. The course was much easier than HPC. The lectures were entertaining. It turned out to be a good primer for other concepts that are covered throughout the Machine Learning specialization, and I’m glad I took it before ML. And practically, it was a nice introduction to working with standard Python tools like NumPy, Pandas, etc. This course was also where I first learned about options.
This class covered a lot of material, including: linear image processing, Hough transforms, feature detection, optical flow, camera calibration, and tracking. I was surprised how powerful classical computer vision algorithms could be. CV was my first introduction to the Kalman filter, which would crop up in other courses as well. My final project for the course was on augmented reality: projecting an object into a 3D video. It was cool to see how this technology works under the hood. CV was one of my favorite classes.
ML is another class with high-quality lecture production. It’s a great overview of supervised learning, unsupervised learning, and reinforcement learning. The assignments are writing-heavy. I really liked that aspect of it, because by writing about the output and behavior of various ML algorithms, it helped me develop intuition about how these tools worked. I had previously taken Andrew Ng’s Machine Learning course, and so I felt well prepared for this class.
The lectures for this course are taught by Sebastian Thrun, the founder of Google’s self-driving car team. Dr. Thrun held office hours for the course as well, which was cool. The material covers basic robotics algorithms, with a focus on robotic vehicles: Kalman and particle filters, search algorithms like A*, PID controllers, and SLAM. The assignments for this course were fun because you get to drive a little robotic actor through scenes. After this class is when I started becoming anxious to wrap up the program.
Spring 2020 was the only term where I took two courses simultaneously. I imagined they would have some overlapping ideas, since they were both stats classes, and neither seemed especially hard. I managed to get an A in both courses, but it was definitely a lot of effort to coordinate the workloads and fit them into my schedule.
Simulation and Modeling for Engineering and Science was an interesting course. It was all about simulation systems: hand simulations, monte-carlo, the Arena simulation language, random variate generation, and input and output analysis. The lectures were entertaining, and there was a lot of material. This course was a little harder than I expected, probably because I didn’t have a strong stats background.
Bayesian Statistics had some overlapping ideas about probability distributions and monte-carlo methods. It was a deep-dive on Bayes Theorem and Bayesian analysis. The material covered Bayes formula, Bayesian networks, OpenBUGS, Bayesian inference, Bayesian computation, MCMC methodology, and more. This course helped me think in Bayesian terms, which is sometimes counter intuitive.
DVA is a broad introduction to data visualization. This was the first course I took where there was a group project. The course touched on data collection, data cleaning, SQLite, data integration, data analytics, Hadoop, Spark, D3, classification, and ensemble methods. I wasn’t crazy about this course. There didn’t seem to be any cohesion between the various ideas, and there was just a superficial coverage of the topics. I did learn how to use D3, though, which was useful.
This course was quite interesting. Most of the hype-generating news in the Machine Learning world is related to Deep Learning, so it was fascinating to learn how these powerful models actually work. The course covered neural networks and gradient descent, optimization of deep networks, convolutional neural nets, pooling layers, PyTorch, bias and fairness, language models, embeddings, transformers, attention, and generative models. This was another one of my favorite classes.
GA has earned a reputation of being difficult, and it is a core requirement. As many students do, I took this as my final course. It was challenging, but I put in plenty of effort and had no issues. It covers dynamic programming, graph algorithms, and NP-completeness all in depth. The material is well organized. The grading is heavily based on three exams spaced throughout the course, but the homeworks do a good job of preparing one for them. It felt rewarding to complete this class, and I was glad to brush up on concepts I hadn’t studied since undergrad.
]]>Scenario: You have an existing IPython / Jupyter notebook with a directory of additional modules, and you’d like to run the project on Google Colaboratory (Colab).
Here’s how to do it.
!pip install <the requirements>
!ls
%cd
to change the working directory to your project’s directory, e.g. %cd drive/MyDrive/path/to/project
importlib.reload
. I’ve read that the %autoreload
magic can work here, too, but I haven’t tried it.spotify-tui
, a Spotify UI for the terminal, but most of the text was not visible in my terminal with the Solarized Light theme!
As per the readme, you can set a theme in ~/.config/spotify-tui/config.yml
. Here’s a theme definition with the Solarized Light color palette:
theme:
active: "42, 161, 152" # current playing song in list
banner: "42, 161, 152" # the "spotify-tui" banner on launch
error_border: "220, 50, 47" # error dialog border
error_text: "220, 50, 47" # error message text (e.g. "Spotify API reported error 404")
hint: "38, 139, 210" # hint text in errors
hovered: "211, 54, 130" # hovered pane border
inactive: "147, 161, 161" # borders of inactive panes
playbar_background: "7, 54, 66" # background of progress bar
playbar_progress: "42, 161, 152" # filled-in part of the progress bar
playbar_text: "101, 123, 131" # artist name in player pane
selected: "42, 161, 152" # a) selected pane border, b) hovered item in list, & c) track title in player
text: "101, 123, 131" # text in panes
const API_KEY = "REPLACE_ME";
// paths to fields to extract from xml
// e.g., ["authors", "author", "name"] maps to "Charles Dickens" in this XML:
// <authors>
// <author>
// <name>Charles Dickens</name>
// </author>
// </authors>
const FIELD_PATHS = [
["title"],
["authors", "author", "name"],
["publication_year"],
["description"],
["num_pages"],
["image_url"], // to display an image from this url, create a formula cell
// with `=IMAGE(X)`, where X is the cell containing this url
];
function bookDataFromISBN(isbn) {
// isbn field must be text instead of number so that leading 0s are retained
if (isbn == "") { return; }
const url = `https://www.goodreads.com/book/isbn/${isbn}?format=xml&key=${API_KEY}`;
const xml = UrlFetchApp.fetch(url).getContentText();
const document = XmlService.parse(xml);
const root = document.getRootElement();
const book = root.getChildren("book")[0];
return [FIELD_PATHS.map(fieldPath => {
return fieldPath.reduce((obj, pathPart) => {
return obj.getChild(pathPart);
}, book).getText().replace( /(<([^>]+)>)/ig, '');
})];
}
This post further explores the Google Sheets Apps Script integration. It shows how you can define a simple templatized web page in a sheet. You may find this useful if you’re building a small prototype, need to quickly put a customizable page on the web, or if you, like me, just like playing with Google Sheets. A web page built in this way is free to host, and is very easy to deploy and customize. That said, I wouldn’t recommend this setup for maintaining a production web application because it’s brittle and non-standard, and, since 2017, Google places a security header at the top of the web page (though the header can be hidden with a browser extension).
Google Apps Script is a powerful low-code solution for building web-apps, and it can do much more than what’s covered in my blog post here. There are other tutorials online about how to create web apps with Google Apps Script. This post instead focuses on using a minimal, generic script, and moving web page customization into the sheet itself. It does this by having the sheet store the dynamic HTML template, and cells that are fed into the template are designated within the sheet.
The toy web page we’re about to build needs some data. Let’s say we’re making a company’s lunch menu that will be displayed on monitors outside the lunch room. First, navigate to https://sheet.new to create a new Google Sheet. Maybe we’ll show the following dynamic information:
A couple notes… Ignore cell A1 for now, that’s where we’ll put the steak sauce, err I mean that’s where we’ll put the HTML template. Note that the cell in B2 contains text data, not an actual date. This is necessary because the script we’ll add later doesn’t know how to format the raw data it receives, so we’ll do that formatting directly in the sheet. You can make a dynamic day of week field by using the formula =TODAY()
and then changing the date format to only display the day of the week.
So note that, using the sheet itself, we can do arbitrary pre-processing of the data before readying it for display. I have organized this sheet by putting labels for the data in column A and the values to be displayed in column B, but that’s incidental. In the next section we’ll see that data can come from any cell.
An HTML template combines static HTML elements with special sentinels denoting variables to produce dynamic HTML output. I created an absolutely minimal templating language here, where template variables are surrounded by {
curly braces}
, and they must refer to cells with data to display. There are plenty of fully-featured templating languages available – Handlebars is a famous one.
As a small example, we could define a template like this:
<div>
<i>Hello</i>, {B2}!
</div>
If the cell B2
in our sheet contained the text World
, the resultant HTML would look like this:
<div>
<i>Hello</i>, World!
</div>
The part of the template that contained {B2}
was replaced with the value from cell B2
in our sheet – nice. So let’s build the HTML template for our menu, referencing the data cells defined in the previous section.
<html>
<head>
<style>
body {
background-color: skyblue;
}
</style>
</head>
<body>
<h1>Today's Lunch Menu</h1>
<h2>{B2}</h2>
<h2><i>{B3}</i></h2>
<h3>Appetizers</h3>
<ul>
<li>{B4}</li>
<li>{B5}</li>
<li>{B6}</li>
</ul>
<h3>Entrées</h3>
<ul>
<li>{B7}</li>
<li>{B8}</li>
<li>{B9}</li>
</ul>
<p>
Thanks for dining at the cafeteria!
Please bus your table when you're finished eating!
</p>
</body>
</html>
This text should go in cell A1, which is where the script will read it from.
Now, open the script editor by going to Tools > Script Editor. Here is the Apps Script script we’ll use to read data from the sheet, combine sheet variables with the HTML template, and serve that HTML in response to a GET request.
function letterToColumnOrdinal(letter) {
// TODO: handle >= AA
return letter.charCodeAt(0) - "A".charCodeAt(0);
}
function dataAtCellAddress(data, cellAddress) {
const row = Number.parseInt(cellAddress.match(/\d+/)[0]) - 1;
const col = letterToColumnOrdinal(cellAddress.match(/[A-Z]+/)[0]);
return data[row][col];
}
function doGet(e) {
const sheet = SpreadsheetApp.getActiveSheet();
const data = sheet.getDataRange().getValues();
const template = data[0][0];
const html = template.replace(/{(.*?)}/g, function(match, p1) {
return dataAtCellAddress(data, p1)
});
return HtmlService.createHtmlOutput(html);
}
The letterToColumnOrdinal
function is just a helper function to take a letter from the cell address and convert it to a number so that we can index into the data
matrix, e.g. C -> 2
. dataAtCellAddress
takes the sheet data and a raw cell address and returns the value stored in that cell. The doGet
function is called in response to a GET
request. It is responsible for extracting data from the sheet and combining the cell data with the template string. When the site’s info or structure needs change, those modifications can happen directly in the sheet, and these 20 lines of Apps Script code won’t need to be touched.
Let’s deploy our new web page. From within the script editor, go to Publish > Deploy as web app.
Then, fill out the dialog form and copy the URL to the web page. Here’s the example page I created: Lunch Menu.
Great! Finally, let’s say the chef was inspired by this blog post to make a new app. So he’s eighty-sixing the mezze platter in favor of a bruschetta crostini. All we need to do is update that cell in the sheet…
…and reload the web page.
And that’s it. We now have a simple, templatized web page that’s easy to change via the sheet.
]]>generation-p
: an interactive art project driven by a genetic algorithm. The interface for this project is a Twitter bot. Once per day, the bot posts a new “individual” – a 32x32 pixel image – from the genetic algorithm’s population.
This project explores the emergence of meaning from randomness. Can a population of images, which begins as noise, evolve into something worthy of attention? Retweets and favorites are a proxy for attention, and so they serve as inputs to the algorithm’s fitness function.
Over time, those images which garner more attention have a higher likelihood of being selected for reproduction, and the idea is that future generations will tend to produce more interesting images. In this way the bot is (very slowly) solving an optimization problem, where the target function is akin to attention-worthiness.
When I began work on this project, the output was textual rather than visual. I quickly pivoted to instead use images. Around early December I was looking at a lot of pixel art, and I loved how the artists were able to eek so much information (and evoke emotion) from so few colors and such little area. This project is partly inspired by pixel art, even if the results don’t look like much of anything. I think nascent patterns in images can be more quickly apparent, and reactions to them can be more immediate, and so images are well suited for this bot.
There are two crossover mechanisms at play. One method is k-point crossover, if the images were first flattened to one dimension. The other method is similar, but based on taking 2D regions from each of the parents. Also, the crossover methods and parameters are themselves inherited and undergo mutation. The selection mechanism is fitness proportional.
Below are examples of how the crossover methods control inheritance from either parent with different parameters.
Crossover with runs of length 2:
Length 8:
Patch crossover with patch-size 2:
Patch-size 8:
The number of individuals per generation, the size of the images, the mutation rates, and other inputs are all hard-coded hyperparameters that are pretty much arbitrary and mostly un-tuned. I suspect that whatever the output converges to will probably just look like noise, since the attention-worthiness fitness function is noisy. Even if everybody who interacted with the bot had a cohesive idea of what they wanted the output to look like, the continuous macaronage of crossover might prevent any “signal” in the image from emerging. But we shall see! I like the idea of a long-running bot like this where the output will change over months and years, because it mirrors the biological evolution we observe playing out in slow, animal time.
Writing this in Clojure was a joyful experience. I like REPL-driven development a lot, and it’s especially useful when experimenting with visual output. I found my crossover implementation to be sort of inelegant, but I think that’s just because my Clojure is rusty. Overall it was easy to get into a flow state with Clojure, and I’m really happy I got to use the language again after having not written anything in it for a long time.
]]>