As 2021 starts to come to a close, and I take the week of Christmas off, I have plenty of time to reflect. All things considered, 2021 was a good year. I celebrated my 25th wedding anniversary (I could probably stop there. That’s a feat that can’t be topped), I watched my kids continue to grow and succeed, and I had a ton of fun making and playing music.
2022 is already shaping up to be awesome. There is some stuff in the works as far as my job is concerned (I can’t talk about that yet, but hopefully soon). And I think I can take my music hobby up to the next level, and that’s super exciting for me.
This year one of my main goals was that I wanted to play guitar on someone’s record. I didn’t get that done. But I did grow in the area of recording and composing. I record somewhat regularly. I’ve learned how to use a DAW fairly well. My mixing skills need work, but I can record something and mix it so it sounds decent. I also write music a fair amount, but I don’t finish much. The flip side of that is that I’ve become pretty good at creating parts and layering parts for music.
If you know me, you know I love analyzing things. After years of studying the theory behind music, I’m now able to analyze a piece of music, determine it’s purpose, and hopefully add something to it that helps it communicate that message a little better. And I LOVE it.
This year I also considered starting something in music. I don’t know what that is. A YouTube channel, a music site, or maybe something else. I did create a not so secret Instagram account for guitar stuff (scottradcliffguitar). It’s fun when I put stuff there, and I’ve slowly become better, but it’s temporary.I imagine 2022 will be more of that. But I think there is a sweet spot in skills of software engineering and musician that could be really interesting.
But overall what I really want is more recording. My goal for 2022 remains the same. Playing on someone’s record. But it also includes releasing some stuff. I don’t sing, so it’ll be instrumental, but still fun for me.
I’m grateful and looking forward to what is next. Here is to a great 2022!
This is the fifth post in a series about building an Elixir library. You can see the other posts here:
And again, I haven’t touched this project in a long time. Revisiting it, and checking the README, it looks like Blockquotes are up next. Should be pretty simple. I opted for just one level of blockquote for now.
Initially I thought that I would need another module like Hyperlink, Bold, or Italics. Turns out I didn’t need a new module. It’s really simple. One function does everything.
def create_blockquote(text) do
replaced = String.replace_prefix(text, "> ", "<blockquote>")
|> String.replace_suffix("", "</blockquote>")
"<p>#{replaced}</p>"
end
Rather than doing some regex capture and then replace, I opted to just replace the beginning and end of the string if it starts with “> “. Looking at this now, I may need to update to also accept a “>” without the additional space. I’m not sure. I’ll look up some markdown docs to see if the space is typically required.
This did require one more change. I needed to update the parse method to react to a line that starts with “>”.
defp parse(text) do
cond do
String.match?(text, ~r/^\#/) -> create_heading(text)
String.match?(text, ~r/^[[:alpha:]]/) -> create_paragaph(text)
String.match?(text, ~r/^>/) -> create_blockquote(text)
end
end
That last line directs to the blockquote code.
While throwing data in the module to check the results, I noticed a bug. If a string has multiple newlines, it doesn’t split properly and breaks the whole thing. So I updated the split function to use a regex that looks for any number of newlines.
def generate(text) do
parsed = Enum.map(String.split(text, ~r/(\n)+/), fn x ->
String.trim(x)
|> parse
end)
{:ok, Enum.join(parsed)}
end
Next up is code formatting. I expect this to take awhile. I see a good amount of manipulation there.
This is the fourth post in a what is a series about building an Elixir library. You can see the other posts here:
Just as I expected, with most of the work done when parsing links was built, italics was pretty simple. I have some duplication that I would like to remove, but it’s not that important yet, and I’m cautious to add abstractions without a solid reason.
What I ended up doing it adding a pipe to create_paragraph and returning the last part of that Tuple. Maybe I could do something better here, but I don’t hate it.
defp create_paragaph(text) do
"<p>#{elem(Hyperlink.convert(text),1)}</p>"
|> Italics.convert
|> elem(1)
end
And for the italics work, I created a new module and defined very similar methods from the Hyperlink module.
defmodule Italics do
def convert(string) do
text = string
|> capture_sections
|> build_emphasis
|> replace_emphasis(string)
{:ok, text}
end
defp replace_emphasis([head | tail], text) do
replace_emphasis(tail, String.replace(text, matcher(), head, global: false))
end
defp replace_emphasis([], string) do
string
end
defp capture_sections(string) do
Regex.scan(matcher(), string, global: true)
end
defp build_emphasis(captures) do
Enum.map(captures, fn x ->
"<em>#{Enum.at(x, 1)}</em>"
end)
end
defp matcher do
~r/_(?<text>[a-zA-Z0-9\s]+)_/
end
end
If you’ve read the parsing links article, this will look really familiar. The methods almost have the same names, and they share similar responsibilities. The one addition here is the addition of a matcher function to hold that Regex for me. I got tired of forgetting to update the second place that used the same regex.
Looking at this, I can see where I could extract it. But right now it just feels like premature extraction. I like opening this file and seeing everything Italics does right in front of me.
And lastly, the tests are really simple.
test "italicizes text" do
assert Bargain.generate("there is some _text_ here") == {:ok, "<p>there is some <em>text</em> here</p>"}
assert Bargain.generate("there is some _text_ here and _here too_") == {:ok, "<p>there is some <em>text</em> here and <em>here too</em></p>"}
assert Bargain.generate("there _is some text here and here too_") == {:ok, "<p>there <em>is some text here and here too</em></p>"}
end
I just realized that the Italics module doesn’t have unit tests, but there aren’t really necessary. That logic is tested well enough.
Next up, bold text. I expect the same sort of path. Pretty simple.
This is the third post in a what is a series about building an Elixir library. You can see the other posts here:
It’s been a minute since I’ve posted one of these updates. It’s due to a mixture of other things getting in the way and struggles with recursion. I thought I understood recursion in Elixir, but apparently not. I learned recursion so long ago that I’ve forgotten most of it. At any rate, I got it, and I think the lesson I learned is that head | tail is the best approach trying to iterate the same thing multiple times.
The problem is pretty simple in theory. Look for any part of a string that begins with [] and ends with (), extract the contents, build a hyperlink with the contents of () as the url, and the contents of [] as the link text. Then replace the []() part with the actual hyperlink. Do this globally.
The trick here is immutability. The string that you are updating must be new every time. I’ll fix that later too.
I’ll just present my working solution.
In, my main Bargain module, I updated create_paragraph to call out to a new module I made called Hyperlink.
defp create_paragaph(text) do
"<p>#{Hyperlink.convert(text)}</p>"
end
And here’s the entire Hyperlink module
defmodule Hyperlink do
def convert(string) do
links = capture_link_segments(string)
|> build_link
replace_link(links, string)
end
defp replace_link([head | tail], text) do
replace_link(tail, String.replace(text, ~r/\[\w+\]\(http:\/\/\w+\.com\)/, head, global: false))
end
defp replace_link([], string) do
string
end
def capture_link_segments(markdown) do
Regex.scan(~r/\[(?<text>\w+)\]\((?<url>http\:\/\/\w+\.\w+)\)/, markdown)
end
defp build_link(captures) do
Enum.map(captures, fn x ->
"<a href='#{Enum.at(x, 2)}'>#{Enum.at(x, 1)}</a>"
end)
end
end
There is some interesting stuff, so I’ll unpack this a bit.
I will start with convert. That will take our markdown string, pull out the segments (the parts in [] and ()), build links for those into a list, and then replace them all before returning the string. Looking at this now, it should return a tuple {:ok, string}. I’m a believer that all public functions should return tuples, but I’ll do that later.
The capture_link_segments function will return a list of of those captures. More precisely, a list of lists. Given the string “This is a link and another“, it would return:
[
["[link](http://google.com)", "link", "http://google.com"],
["[another](http://google.com)", "another", "http://google.com"]
]
And I just found a bug. That regex doesn’t handle multiple words in the link text.
Then on to build the link with build_link. Pretty simple here. Map over the list and create and actual HTML link.
Now the interesting part, and the part that gave me the most trouble. Recursion.
The tricky part is the string of text. We need to manipulate this, but Elixir likes immutability. I went through lots of very messy iterations of this. I won’t list them here, but they are in the commits on GitHub. What I finally had to do is drop back and review recursion in Elixir and write recursion code outside of this project, so I could focus on the solution. Dave Thomas explains this really well in Programming Elixir, a great resource. I went back to that book for review.
Turns out the solution is pretty simple. Elixir has a way of making complicated things simple, but you first need to understand the complicated thing.
By using head | tail I was able to constantly iterate of the collection until it was empty. And by passing in a string as the second parameter, I was able to constantly build a new string until I was done. The meat of the recursion is in the first replace_link([head | tail], text) function. The match to just return happens in replace_link([], string).
As long as the first parameter has something in the list, it will fire the string replacement operation. If it’s empty, it just returns the string. It took me days to get here. Those days are just a couple of hours here and there, but still, a very long time. Hopefully, I’ll remember this pattern going forward.
Next up is fixing the bugs I found, then italics, bold, etc… I should be able to reuse some of this recursion logic to complete that.
Update: The fix for multiple words was simple. I needed to update the Regex to accept word boundary or whitespace. ~r/\[(?<text>[\w\s]+)\]\((?<url>http\:\/\/\w+\.\w+)\)/. I still need to address the url part of th regex
I work on an app that logs a bunch of data. Typically, I wouldn’t test logging, but we have some issues where bad pattern matching is causing some 500 errors because it can’t handle the responses we are getting.
After some poking around, I ended up on a cool way in ExUnit to test for specific log messages. But there is one major issue that tripped me u[]
The module is ExUnit.CaptureLog and allows you to pass in a function and check that the log has the message you expect.
On the surface, it’s pretty simple. assert capture_log(fn -> Logger.error(msg) end) =~ msg That will assert that Logger.error(msg) actually logs msg.
Just replace Logger.error(msg) with the function that performs the logging and your all set. Mine looks like this because I know if it got to logging something that contains webhook_received, the code executed properly: assert capture_log(fn -> Persistence.MessageQueue.log_message(event_data) end) =~ "webhook_received"
Unfortunately, I was getting a success message when it clearly wasn’t passing. I really don’t know why. What I do know is that the log setting in config directly effects this, and defaults to :warn. By setting that to :info, messages start to flow through when testing, and the assertions start to actually pass/fail accurately.
You may have already noticed that setting the log messages to info makes for a really noisy test suite. But, there is a way around that also.
I haven’t gotten to it yet, but @tag :capture_log will allow you to define which tests should report logs and which ones shouldn’t.
This forum thread was super helpful.
This is the second post in a what is a series about building an Elixir library. You can see the first post here -> Creating an Elixir Library.
Now on to adding paragraphs. Which should be pretty easy. But before that, I need to add one change to the heading parsing code. I want to make sure it handles any number of words.
To do this, I need to add one line to create_heading. Just Enum.join(tl, " ") which just takes everything in the tail and combines them with a space separator.
defp create_heading(text) do
[hd | tl] = heading_text(text)
level = String.length(hd)
"<h#{level}>#{Enum.join(tl, " ")}</h#{level}>"
end
A couple of expectations to make sure everything works. Just add a couple of cases with multiple words.
test "Generates headings" do
assert Bargain.generate("# Heading") == {:ok,"<h1>Heading</h1>"}
assert Bargain.generate("## Heading Two") == {:ok,"<h2>Heading Two</h2>"}
assert Bargain.generate("### Heading Two Three") == {:ok,"<h3>Heading Two Three</h3>"}
assert Bargain.generate("#### Heading") == {:ok,"<h4>Heading</h4>"}
assert Bargain.generate("##### Heading") == {:ok,"<h5>Heading</h5>"}
assert Bargain.generate("###### Heading") == {:ok,"<h6>Heading</h6>"}
end
And now all headings work, but we still only handle ones with pound signs. I haven’t decided if I want to handle other versions.
Now on to paragraphs. This seems super simple, but I ran into a few issues as I started to refactor to make sure multiple things could be handled at the same time. For example, a heading followed by a couple of paragraphs.
To start I went with the ridiculously simple TDD version of just returning exactly what I want.
def generate(text) do
case String.starts_with?(text, "#") do
true -> {:ok, create_heading(text)}
false -> {:ok, create_paragaph(text)}
end
end
defp create_paragaph(text) do
"<p>#{text}</p>"
end
Here I just simply add a false clause. If the string doesn’t start with a pound sign, assume it’s a paragraph and return whatever that is.
Simple test.
test "Generates paragraphs" do
assert Bargain.generate("This is a paragraph") == {:ok, "<p>This is a paragraph</p>"}
end
Now I needed to handle multiple lines. A little tricky here. I wasn’t sure how to split one string into multiple lines. In Ruby I can call lines on a string and get what I need, so I looked for something similar. Turns out it’s just easier to split on something and enumerate through those. That was the approach I took.
def generate(text) do
parsed = Enum.map(String.split(text, "\n"), fn x ->
String.trim(x)
|> parse
end)
{:ok, Enum.join(parsed)}
end
defp parse(text) do
cond do
String.match?(text, ~r/^\#/) -> create_heading(text)
String.match?(text, ~r/^[a-zA-Z]/) -> create_paragaph(text)
end
end
You can see in the generate function that I split the string on any newline (technically not correct for markdown, but I’ll return to that), remove leading and trailing spaces, and pass it in to be parsed. The use of map here is important because it returns a list. We need that list so we can return the string we have built up.
parse is interesting. I almost always reach for case and do some pattern matching, but I struggled getting some level of regex pattern matching working. The reason being is that the regex returns true/false. That only gives me two possible matches, like in the previous version of generate. I’ll have plenty more than that.
So the solution is cond. With cond I can perform several different matches and react to them accordingly.
An interesting side note. Elixir has character classes you can use. Considering that a paragraph will likely start with a letter, because a number will probably indicate a list, I used [:alpha] in place of [[a-zA-Z]], but it acted strangely. – I just realized my problem. I’ll go back and fix it. [:alpha] didn’t always match because of a syntax error. It should be [:alpha:] (see the trailing :).
The last thing I want to point out is the use of join at the end of generate. Since we get a list back, something like this ["<h1>Heading</h1>", "<p>This is a paragraph</p>"], we need to make that one big string to return.
And finally some tests to make sure I stitched everything together correctly.
test "Generates paragraphs" do
assert Bargain.generate("This is a paragraph") == {:ok, "<p>This is a paragraph</p>"}
assert Bargain.generate("This is a paragraph") == {:ok, "<p>This is a paragraph</p>"}
assert Bargain.generate("This is a paragraph\nAnd another") == {:ok, "<p>This is a paragraph</p><p>And another</p>"}
end
test "Generates a heading and paragraphs" do
test_string = "## This is a heading
With a paragraph
and another paragraph"
assert Bargain.generate(test_string) == {:ok, "<h2>This is a heading</h2><p>With a paragraph</p><p>and another paragraph</p>"}
end
A couple of things here. In Generates paragraphs, I wanted to make sure I tested more than one paragraph at a time. And in Generates a heading and parapgraphs, I wanted to make sure I could test a more complete markdown string with a heading and a couple of paragraphs at the same time.
Next up is links.
I’ve been writing Elixir for a while professionally. I’ve worked on a couple of production apps. Mainly umbrella apps with just a little bit of Phoenix for API end points. And that’s a ton of fun. But I want to do more. So I decided to just write a bunch of stuff in Elixir. Basically all of my sites/software. A good place to start is a simple library. It’s a good exercise, and a good opportunity to learn things that I haven’t used much, and then share what I learn.
I’m writing a markdown library called Bargain. The readme has a list of features I’m going to implement as part of the generator. I’ll write something up each time I learn something new or implement a new part. Starting now.
I started with generating headings. Specifically the # version. In true TDD style, the first passing examples were crazy simple.
def generate(text) do
case String.starts_with?(text, "#") do
true -> {:ok, create_heading(text)}
end
end
defp create_heading(text) do
"<h1>#{heading_text(text)}</h1>"
end
defp heading_text(text) do
String.trim(text)
|> String.split
|> tl
|> Enum.join(" ")
end
It just takes some text, checks to see if it starts with heading syntax. If it does, push it to create the heading.
I originally had the heading_text syntax in the create_heading function, but decided to break it out into a function that always returns the text of the heading. It just cleans up the string and splits it. tl is probably the most confusing part, it breaks it up into head and tail, tl gives me everything except for the first part of that List, and then just join the remaining parts into a string of text.
And finally, the obvious part of just returning h1 explicitly.
The first test looks like this:
test "Generates HTML" do
assert Bargain.generate("# Heading") == {:ok,"<h1>Heading</h1>"}
end
Pretty simple. I just wanted to make sure it was wired properly and returning a tuple.
The logic for returning this for all headings is a little more interesting.
The tests.
test "Generates headings" do
assert Bargain.generate("# Heading") == {:ok,"<h1>Heading</h1>"}
assert Bargain.generate("## Heading") == {:ok,"<h2>Heading</h2>"}
assert Bargain.generate("### Heading") == {:ok,"<h3>Heading</h3>"}
assert Bargain.generate("#### Heading") == {:ok,"<h4>Heading</h4>"}
assert Bargain.generate("##### Heading") == {:ok,"<h5>Heading</h5>"}
assert Bargain.generate("###### Heading") == {:ok,"<h6>Heading</h6>"}
end
I could have had a separate test for each heading, but this is fine.
And the updated code.
def generate(text) do
case String.starts_with?(text, "#") do
true -> {:ok, create_heading(text)}
end
end
defp create_heading(text) do
[hd | tl] = heading_text(text)
level = String.length(hd)
"<h#{level}>#{tl}</h#{level}>"
end
defp heading_text(text) do
String.trim(text)
|> String.split
end
The generate function doesn’t change, but create_heading gets more complex
Instead of calling tl in heading_text, I opted to call hd | tl in create_heading so I can get the pound signs for heading level and the text of the heading in one call.
Now I can just call String.length to get the heading level, and just display the rest. You may notice that I’m not handling multiple words in the heading. tl will be a List.
I’ll start on that next. After that I’ll start into paragraphs.
Sounds a bit bold, and maybe an overreaction, but I think it’s true. Without Elixir, I was thinking of doing something else.
I’ve been pretty vocal about my distaste for the current state of front-end development. And I was getting to a point that if front-end dev was the future of software development, I wasn’t interested. I was pretty close to just calling in quits and doing something else.
Then I got recruited, started a new gig, started doing Elixir seriously, and developed a new love for making software again.
I sure hope things like Elixir are the future of programming. And keep in mind, if you don’t like a particular portion of software development, you can just concentrate on something else. Regardless of what someone else tells you.
If you log into servers regularly, you may run into an issue with a failed RSA key verification. This just means that something changed in the handshake between your machine and the server. If you are sure everything is okay. And that’s important. Make sure the server you are logging into is actually the server you think it is! Then the fix is simple. The error message will give you the failed key and even the line it’s on. Open the file ~/.ssh/known_hosts, turn wrapping off so you see one long line at a time. Go to the line in question. And then delete that line. SSH into the server again and save the new RSA key.
This seems really simple, but I was confused the first time this happened. The error is usually because DevOps changed something. But luckily the fix is simple.
I’m working on my second production Elixir system and ran into something that’s a bit of a no-brainer in Ruby. I needed yesterday’s date. I’m working on an integration that needs to send a date range as start date and end date for an API request. That date is always yesterday and today. Basically, give me everything from yesterday until now.
Crazy simple in Ruby. Without Rails: Date.today.prev_day and with Rails: Date.yesterday.
I tried typing that in Elixir. Doesn’t work.
Elixir’s date library is really good, but doesn’t have something for yesterday. And this is rare, but the docs don’t really tell you how to do that.
The solution is to create a date and pipe into the add function with -1 as the parameter.
We can just use utc_today() to get today’s date and use that for our pipe.
Date.utc_today()
|> Date.add(-1)