Searching Splunk: What I Learned from the SANS Holiday Hack Challenge 2021

Christmas came early this year. Most notably, in the form of the annual SANS Holiday Hack (otherwise known as KringleCon). I wasn’t sure what to expect, but I knew it would involve lots of learning and some strong holiday spirit. I wasn’t wrong on either of those fronts.

The first challenge I stumbled across was on the topic of Splunk. Splunk is a technology I had been wanting to try out, so my first holiday hack was off to a great start.

Disclaimer: If you find any of this information to be wrong, please shoot me a note! I’m always looking for learning moments :)

Task 1: Capture the commands Eddie ran most often, starting with git. Looking only at his process launches as reported by Sysmon, record the most common git-related CommandLine that Eddie seemed to use.

We were given an example search to sort results on various parameters:

This search looks at Eddie’s process launches from the Command Line field as reported by Sysmon. It adds up how many of each command it sees and then sorts from most common to least common. The question specifically asks about git-related commands. As we can see, git status is at the top of the list (and is the top git-related command) so it is the winner.

And there we have it! First task down and a I was now a little more comfortable with the Splunk search feature. Off to the next one!

Task 2: Looking through the git commands Eddie ran, determine the remote repository that he configured as the origin for the ‘partnerapi’ repo. The correct one!

The example search below was given to us to view all events. 28,304 events definitely calls for some creative searching. Luckily, that’s what Splunk is made for.

In order to find the answer to task two, I needed to look specifically at git commands Eddie ran. Based on the fields given, the answer probably lies in the CommandLine field. Upon looking at all CommandLine entries and searching for the word “git” and “remote”, we shaved our event list down to 8 entries.

Within the 8 events, there are two places that a remote is added: event 5 and event 8.

The remote added in event 8 is deleted in event 6. Therefore, the newest one added should be the right answer.

And it looks like “git@github.com:elfnp3/partnerapi.git” was accepted as the correct answer. Now that I had some event searching down, I felt ready to move onto the next task.

Task 3: Eddie was running Docker on his workstation. Gather the full command line that Eddie used to bring up a the partnerapi project on his workstation.

The first thing I did was create a search on the entire CommandLine field that found “partnerapi” and “docker” in the command. This gave me 4 events to look at.

The first two were “git add Dockerfile…” commands, then “docker compose up”, and finally “/usr/bin/python3 /usr/bin/docker-compose up”. I didn’t know much about Docker commands, so I did some investigation. Docker documentation (dock-umentation?) stated that Docker compose up starts a docker application.

I interpreted “bring up” in the task question as “run” and guessed that “docker compose up” was the answer. With 100% transparency, I wasn’t 100% sure this was the answer — I still had “git add Dockerfile…” in the running. I couldn’t find much information on that command, and it looked to be adding the docker file to Github, so I went with compose.

Accepted! Another task marked off the list and I was on a roll. On to the next task!

Task 4: Eddie had been testing automated static application security testing (SAST) in GitHub. Vulnerability reports have been coming into Splunk in JSON format via GitHub webhooks. Search all the events in the main index in Splunk and use the sourcetype field to locate these reports. Determine the name of the vulnerable GitHub repository that the elves cloned for testing and document it here. Inspect the repository.name field in Splunk.

The amount of information given was a little daunting, but breaking it down into parts made it much clearer:

Search all events > Search all webhook events > Search all Github-related webhook events > Search all Github-related webhook events that mention “repo”.

I utilized another example search that was given to me that searched webhook-related events using “sourcetype=github_json”:

This search narrowed the number of events significantly. I noticed throughout many of the events, the field “https://github.com/elfnp3/dvws-node/security/code-scanning/[INSERT NUMBER HERE]” was mentioned along with “code injection”.

I tried submitting both “dvws-node”, “elfnp3/dvws-node”, and “https://github.com/elfnp3/dvws-node” but all were rejected. I took a look at elfnp3 on Github and found they had cloned “dvws-node” from Snoopy Security. I then submitted “https://github.com/snoopysecurity/dvws-node” which worked. Fourth time’s a charm!

Task 5: Santa asked Eddie to add a JavaScript library from NPM to the ‘partnerapi’ project. Determine the name of the library and record it here for our workshop documentation.

The key words here are NPM and partnerapi. I added those as search terms which brought the event number down to 5.

The first event (chronologically) was an install command. The task said find the library that was added. Adding via NPM = install. I was further confident this led me to the right package due to the name relating to Javascript — “holiday-utils-js”.

Sure enough, that did the trick! Now onto task 6…

Task 6: Another elf started gathering a baseline of the network activity that Eddie generated. Start with their search and capture the full process_name field of anything that looks suspicious.

This is the search that was given to start with:

Breaking it down, we can see it’s searching Sysmon events under the user “eddie” that does not involve the loopback address () and isn’t using SSH, DNS, HTTP, and HTTPS ports. It then aggregates each seen destination IP + destination port combination into how many times it was found.

This gives me 3 events to work with, however only two unique IP/port combinations.

I expanded each one and clicked “View events” which gave me more info to work with. The process name of the first one was “/usr/bin/git”. It looks as though Git was establishing an outbound network connection. The second IP/port combination was establishing an outbound network connection as well, but the process name was “/usr/bin/nc.openbsd”.

With a quick Google search, I found that OpenBSD is a built-out version of netcat that uses TCP and UDP to read and write data across network connections. It is commonly used in reverse shell attacks.

This definitely looks like the suspicious activity we are trying to find. I copied the process name into the answer section and with that, task 6 was complete.

I have yet to finish writing out task 7 and 8, but I guess you could say I’m enthusiastic about getting this article into the cyber void. I just wanted to take some moments to say what I learned from all this:

1. Splunk is an absolute powerhouse. The amount of data it can sift through faster than lightening speed blew me away — and definitely made me realize why Splunk is so highly thought of within the security community.
2. I surprised myself at how quickly I was able to pick up searching in Splunk. I have no doubt that the efficiencies of Splunk searching are an infinite rabbit hole that I look forward to burying myself in. But as a beginning Splunker without previous experience, I’d give myself a pat on the back or two.
3. SANS did an amazing job with their holiday hack. You can tell just how dedicated the creators were to making it a great experience. Thank you to those who contributed. I’ll see you next year!