GNU/Linux nerd and linuxgamer.
65 stories
·
1 follower

Stop deploying web application firewalls

1 Comment

I wanted to write this because I don't hear enough real people discouraging the use of Web Application Firewalls (WAFs). Probably because the search results for "Web Application Firewall" are all written by WAF vendors. Anyone reading just that could conclude that WAFs are a good idea. I'm here to offer another perspective, after having suffered through using a WAF for two years.

Web Application Firewalls were created early in the Internet's history, especially popularized by the ModSecurity project in 2002. WAFs essentially work by intercepting every single HTTP request (and sometimes responses too) and evaluating several hundred regular expressions over the URI, headers, and body, sometimes aided by machine learning. If the request kinda looks like SQL, shell code, etc., the server may block your request.

In the infancy of the cybersecurity field, WAFs seemed like a good idea. HTTP requests were tiny, infrequent, and mostly contained mundane form data. But today, WAFs have overstayed their welcome in the security toolbelt. There are better techniques you can use that make even the most advanced WAFs entirely obsolete.

WAFs have Horrible Performance

Since WAFs run hundreds of regular expressions on every request, you may ask, "isn't that super inefficient?" Yes, very.

WAFNo WAF
Average time taken to upload 9,462 text files7.364.55
Average requests per second12852079
Number of requests blocked erroneously50
Peak nginx CPU during trial73%8%
Specifics about the benchmark
The easiest way I know to get modsecurity + CoreRuleSet installed is through ingress-nginx, which I've installed in a Kind cluster.
# https://kind.sigs.k8s.io/docs/user/quick-start/
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 32080
    hostPort: 32080
    protocol: TCP
  - containerPort: 32443
    hostPort: 32443
    protocol: TCP
EOF

# https://kubernetes.github.io/ingress-nginx/user-guide/third-party-addons/modsecurity/
helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=NodePort \
  --set controller.service.nodePorts.https=32443 \
  --set controller.service.nodePorts.http=32080 \
  --set controller.ingressClassResource.default=true \
  --set controller.allowSnippetAnnotations=true

For the test, I'll be uploading files to MinIO using these values:

replicas: 1
mode: standalone
resources:
  requests:
    memory: 512Mi
persistence:
  enabled: false
rootUser: rootuser
rootPassword: rootpass123
buckets:
  - name: bucket1
    policy: none
    purge: false
ingress:
  enabled: true
  hosts: [minio-waf.localhost]
  annotations:
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      Include /etc/nginx/owasp-modsecurity-crs/nginx-modsecurity.conf
      SecRuleEngine On
      # Even the core rules are ridiculous, blocking PUT requests, certain content-types, or any body with "options" in it
      SecRuleRemoveById 911100 920420 921110
helm upgrade --install minio minio/minio -f values.yaml -n minio --create-namespace
helm upgrade --install minio-waf minio/minio -f values-waf.yaml -n minio-waf --create-namespace
# Verify the WAF is working (should get a 403)
curl 'http://minio-waf.localhost:32080/?q=../../etc/passwd'

We'll be uploading just the "Documentation" folder of the v6.6 Linux Kernel, which contains 9462 files for a total of 65MB.

curl -LO https://github.com/torvalds/linux/archive/refs/tags/v6.6.zip
unzip v6.6.zip 'linux-6.6/Documentation/*'

Configure the minio client:

# You may need to add these hosts to /etc/hosts
export MC_HOST_nowaf='http://rootuser:rootpass123@minio.localhost:32080'
export MC_HOST_waf='http://rootuser:rootpass123@minio-waf.localhost:32080'

Run the benchmark (5 times each):

time mc cp -r linux-6.6/Documentation/ waf/bucket1/
time mc cp -r linux-6.6/Documentation/ nowaf/bucket1/

In addition to slowing down every request, you also need significant additional RAM for buffering requests. Since not a single byte in the buffer can be flushed to the backend server until the WAF completes its analysis, you need several gigabytes of RAM to store request bodies. Servers like nginx buffer requests by default, but enough large concurrent requests (like pushing a container image) can make a buffering web server run out of RAM. When using a WAF, every server becomes a buffering web server, which is simply incompatible with many types of applications.

I know computers are fast and hardware is cheap, but we shouldn't be spending that kind of CPU and RAM on WAFs unless they're a really effective security tool. But they aren't, as you'll see next.

WAFs are Easily Bypassed

WAF vendors and attackers are locked in a constant arms race, but it seems attackers are much better armed. How could they not be? Many of the attacks that a WAF purports to block involve complex grammars like SQL, shell code, and entire programming languages. They often include comments, character escaping, encoding issues, and more oddities. These oddities mean that attackers always have a significant advantage and can typically bypass any WAF rule if they are clever enough.

For example, you might think Log4shell is pretty easy to catch: just check for ${jndi, right? Unfortunately, Log4J supports nested "lookups", including ones that convert letters to upper/lower case like ${lower:J}

That means an attacker can insert an arbitrary number of nested lookups around each letter and still perform the attack, like this: ${${lower:J}ndi:.... This lead CloudFlare to say "WAF vendors need to be looking at any occurrence of ${ and treating it as suspicious", which is just another hilarious example of how WAFs can never live up to the expectations placed on them.

I just discussed the fairly simple grammar that is Log4J Lookups, but you can imagine how many more evasion tactics you could use in a language as complex as SQL or PHP, especially when considering encoding tricks. For an in-depth description of specific WAF bypass techniques, check out this awesome post.

Another way to bypass a WAF involves just padding your attack string to appear >8KB or so into the request body. Like I mentioned in the section on performance, request bodies must be buffered into RAM for analysis, so WAFs must choose some cut-off point to avoid spending infinite CPU and RAM on a single request. For some WAFs like AWS's, that cutoff point is around 8KB. So if you just put 8192 innocuous characters before your Log4Shell attack string, you've rendered the WAF worthless.

WAFs are an Attack Vector

In 2019, CapitalOne experienced a breach of 100 million credit applications that was allegedly caused by a WAF misconfiguration. The attacker allegedly tricked the WAF into sending requests to the EC2 Metadata Service, which handed out a credential that allowed reading sensitive files from S3.

While this is just one example, it illustrates the curious fact that WAFs actually have a large attack surface.

Most WAFs are giant, complex codebases that are usually closed-source and written in memory-unsafe languages. Since they're expensive "enterprise" products, companies stuff them full of unnecessary features to make them stand out more than competitors. All of this adds up to make WAFs yet another example of a dangerous "security" tool, just like SolarWinds.

No security officer would approve taking such a risky piece of software, putting it directly on the internet, making it parse mountains of untrusted input, and giving it access to all your backend servers, logging infra, SIEM, alerting systems, and even JIRA for some reason UNLESS it's covered in security buzzwords and costs 5-6 figures per year.

Somehow, companies that sell security products have gotten a pass on implementing foundational security principles like secure by default, secure by design, attack surface reduction, and the principle of least privilege. Don't let them keep getting away with that.

WAFs have a High False Positive Rate

Over the last twenty years, open-source WAF rulesets have expanded considerably to detect more-recent types of attack. Apparently all those proprietary WAFs are doing the same. That means there are more and more possible strings that could trigger a WAF to block your request. If you want to write a comment on an article discussing Log4shell, you might be blocked for including the string ${jndi in your comment. So naturally the false positive rate continues to rise with every new rule, and it's already quite high based on my experience maintaining a giant list of ModSecurity rule exceptions.

So-called "next-generation" WAFs claim to solve this problem by looking at multiple requests or by using IP reputation systems. While these can improve false positive rates, they can never truly solve the problem. In some ways, less false positives can increase the impact of particular false positives since neither users nor support teams have a clear procedure for fixing it. CloudFlare's algorithm can randomly decide to block you and you will have no recourse. Imagine that happening to someone less tech-savvy.

This is the classic problem with using an outdated security tool like a WAF: defenders have to configure the tool absolutely perfectly to be safe and avoid false positives, but attackers just need to find a single weakness. Those are horrible odds. You should use alternatives that don't require perfection from imperfect humans.

Alternatives to WAFs

Since WAFs are resource-hungry, inneffective, unsafe, and noisy, how do I convince an auditor to not make me use one? The technical term would be to use "compensating controls", but that sounds like such a weak term to describe the powerful and simple alternatives to WAFs I'm about to describe:

  • Isolation: Isolation involves ensuring that a breach in one component can not affect the rest of the system, and there are many technologies that provide isolation.
    • Browsers do this by executing all code inside special sandboxed processes that don't have carte blanch access to cookies, saved passwords, other tabs, etc. Imagine how slow the web would be if every piece of JavaScript needed to be analyzed by hundreds of regexes before being executed!
    • Microservices are designed with isolation in mind, but you can also do it in a monolith with a variety of libraries and languages.
  • Immutability: Entire classes of attack can be eliminated by removing a few assumptions, like having a readOnlyRootFilesystem, a package manager that requires rebooting, or append-only/immutable backups.
  • Static Analysis: SQL injection has a miracle cure called "prepared statements". The problem is that devs forget to use them. Static analysis checks in a CI pipeline can all but ensure that zero SQL injection vulnerabilities are in your codebase, at which point there is no need for any SQL injection WAF rules. No, "defense in depth" is not a valid excuse to use a WAF anyway, because it provides no real defense! Like surrounding Fort Knox with an army of guard guinea pigs.
  • Capability-based security: Not every API endpoint needs to have unrestricted read/write access to your entire database and file system, but that is the normal way people build APIs today. By using capabilities, you can express exactly that "GET /api/v1/books" only needs read access to the "books" table. Or that "POST /api/v1/imageupload" needs write access to a specific folder, but doesn't need the ability to spawn processes.

Now I'll admit these ideas are quite broad; you'll need to adapt them to your particular app. WAF vendors offer a one-WAF-fits-all fantasy that I can't match. But these secure-by-design strategies are the way that the security industry needs to be heading. Unfortunately, it's a lot harder for the security industry to profit off of design-based techniques, so don't hold your breath.

Read the whole story
psyq
387 days ago
reply
We can but wish. I don't think in the world of enterprise, where snakeoil is so easily sold, things like "just architect your application properly" will have much traction. Otherwise we wouldn't be in the mess we are.
Switzerland
Share this story
Delete

Borders, Microscopes, and Extinguishing the Ego

1 Comment

First I gotta plug our upcoming retreat.

On April 21-23, 2017 we’re having another Zen retreat at Mount Baldy. Zen retreats are a great way to deepen your practice and get beyond our usual ways of relating to ourselves and others.

I practiced zazen at home by myself for over a decade before I attended my first multi-day retreat with Nishijima Roshi in Shizuoka, Japan in the early 1990’s. In retrospect, I should have sold a guitar or something for the chance to attend one of Katagiri Roshi’s retreats in Minneapolis back when I lived in Ohio. But I was not very smart then.

Now meditation retreats are everywhere. But ours are the best. They’re neither too hard nor too easy, neither too full of ceremony nor too self-consciously devoid of ceremony. We don’t talk your ears off, nor do we leave you to stew in your own juices too much. There’s yoga to help your sore legs and beautiful scenery to remind you what it’s like outside the city.

We’re probably going to cut back on having so many retreats, so get in while you can.

And speaking of ego and the Buddhist idea of no self, here’s a question I got via email:

“Hi Brad. I have a simple question for you that has got me thinking. Through meditation practice, do we ever eventually truly extinguish the ego or do we learn to rise above it by seeing it for what it is (empty) through continuous presence? The Heart Sutra line ‘Form is emptiness. Emptiness is form’ seems applicable if the latter is the case, correct?”

Here’s my answer, embellished for the blog:

I know the traditional metaphor of putting out a fire has been used for thousands of years by Buddhists, but I feel like the phrase “extinguish the ego” is not quite right.

The ego is a mental construct that has some usefulness. Just like “The United States” is a mental construct that has some usefulness.

The ego (or self) is an imaginary border we draw around a group of inter-related aspects of our experience, like we draw an imaginary border around what we call the USA.

Marijuana is now fully legal in California, but if you try to drive across the border at El Chaparral near Tijuana with your mota, you could get in trouble. Likewise, you can buy Viagra at any drug store in Mexico, but bringing it back up the I-5 freeway into California is a problem. Human beings can’t cross that border without being inspected by other human beings. But if a lizard wants to chase a grasshopper across it, other lizards don’t get too fussed. The border is a concept we humans carry in our collective imagination. Yet you can’t really say the border doesn’t exist.

The fire that those ancient Buddhists talk about is the fire that arises because we make mistakes about the nature of ego/self and then get all hot and bothered about things that aren’t actually real. That fire may get (metaphorically) extinguished, but not the ego. Not exactly anyhow.

We think we are what we think we are. But we’re not what we think. At least not entirely.

Because we make mistakes about who we are, we end up worrying about problems that aren’t really problems. Some of us dedicate our entire lives to worrying about problems that aren’t really problems. That’s when it feels like a fire.

Like the imaginary border between the American State of California and the Mexican State of Baja California, the imaginary border we call “self” does have some utilitarian value. I don’t want to get into a big debate about immigration here. But I think people on all sides of the arguments understand the basic value of having national borders. It’s a way to keep conflicts between human cultures from getting out of hand. Even though things still do get crazy sometimes, the concept of a border generally keeps things more peaceful than they’d be if we didn’t have that concept.

Maybe one day human beings won’t need borders. I certainly hope we get to that point. But I know I won’t be alive to see that happen. And I know that if someone tried to force the whole world to open all of its borders before people were ready for it, the results would be catastrophic.

The same may be true with the border called “self.” It’s just as arbitrary and imaginary as any national border. But understanding even that much is difficult. And trying to cross that border before you’re ready can be damaging. That’s why we take this very slowly.

We certainly appear to be eternally separate individuals. I do not have access to anyone else’s memories. I can’t know anything about your internal life unless you tell me. If I smack you on the face, you feel it on your cheek and I feel it on my hand. There seems to be an essential difference between me and the outside world.

It would be idiotic to claim there was no difference at all. There is a difference. And yet, with a bit of meditation practice you begin to see that this difference that seems like such a huge, insurmountable barrier is actually very small.

It’s like you’ve spent your whole life looking through a microscope and then you take your eye away from the eyepiece.

If that actually happened, what you’d see couldn’t fail to be shocking. And yet it’s not as if anything has changed except your perspective. What you saw through the microscope did not become extinguished. It’s still there and you can still see things that way by just putting your eye back on the eyepiece.

All that’s changed is that you’ve seen how small the thing you’d spent your life looking at really was all along.

* * *

Here’s another plug! You can register for the Dogen Sangha Los Angeles Spring Retreat April 21-23, 2017 at Mt Baldy Zen Center right now! 

Led by Brad Warner, this three-day intensive retreat will focus primarily on the practice of zazen. Morning chanting services, work periods, and yoga (led by Nina Snow) will round out the daily activities. The program will also feature lectures by Brad, as well as the opportunity for dokusan (personal meetings).  Participants will be able to take advantage of this beautiful location for hiking during free periods.

Click for the registration form, practice schedules and more!

* * *


DON’T BE A JERK
is now available as an audiobook from Audible.com as are Hardcore Zen and Sit Down and Shut Up and There is No God and He is Always With You!

* * *

I have a YouTube channel now! Check it out!

***

And here’s my Patreon page!

***

 

Check out my podcast with Pirooz Kalayeh, ONCE AGAIN ZEN!

 * * *

I’ve got a new book out now! Stay up to date on my live appearances and more by signing up for our mailing list on the contact page

UPCOMING EVENTS

September 7-10, 2017 Retreat in Finland

September 15-20, 2017 Retreat at Domicilium, Weyarn, Germany

September 22, 2017 Talk in Munich, Germany

September 23, 2017 Retreat in Munich, Germany

September 24-29, 2017 Retreat at Benediktushof, near Wurzburg, Germany

October 1-4, 2017 Retreat in Hebden Bridge, England

ONGOING EVENTS

Every Monday at 7:30pm there’s zazen at Angel City Zen Center (NEW TIME, NEW PLACE!) 1407 West 2nd Street, Los Angeles, CA, 90026 Beginners only!

Every Saturday at 10:00 am there’s zazen at the Angel City Zen Center (NEW PLACE!) 1407 West 2nd Street, Los Angeles, CA, 90026 Beginners only!

These on-going events happen every week even if I am away from Los Angeles. Plenty more info is available on the Dogen Sangha Los Angeles website, dsla.info

* * *

I make my living mostly from your donations. If you find a little bit of truth in what I’m saying remember that even a small donation helps. Your donations are my major means of paying my rent. Thank you!

 
Read the whole story
psyq
2794 days ago
reply
Brad is firing off a broadside of fantastic similes today.
Switzerland
Share this story
Delete

A TAlE OF TWO TALES: HORIZON ZERO DAWN & THE LEGEND OF ZELDA: BREATH OF THE WILD

1 Comment
I've not yet finished Horizon Zero Dawn or Breath of the Wild, but I've been sitting on my tongue for the last few days. There's so much I want to say about them - both in their own right, and in comparison to one another. 

And now? And now I can "park" my "waggler" no longer.

You see, I started playing the new Zelda immediately after playing Horizon Zero Dawn. I was startled by the similarities between them - literally startled; I fell off the sofa, and had to hide beneath the dining table for ten minutes, chittering.

Both feature protagonists who wield a bow and melee weapons. Both are set in a vast open world. And both require the player to gather materials and craft things. 

And yet, while the basic ingredients are the same, and they are essentially the same recipe, they present them in vastly different ways. Where Breath of the Wild takes its ingredients and prepares a dish with great care, in which every ingredient has a place - and the space - to contribute, Zero Dawn just chucks them all in a mixer, pours the resultant slop into a bowl, and grates a whoopee cushion over the top.
IMPORTANT BIT
For the record, before I came to Zelda, I was enjoying Horizon Zero Dawn. It's an unquestionably beautiful game, and though it is fundamentally striving to be photorealistic, there's some really quite lovely art design going on. Also, a lot of dull forests and mountains, which look like all the other forests and mountains in all the other games.

Still, as much as I sort of like it, there was also much about Horizon Zero Dawn which was getting right on my tit-tays. Specifically, it was to do with the way Horizon Zero Dawn tried to tell its story.

At its core, it's actually a decent enough little tale, playing out in a compelling world - set years after an unspecified apocalypse, where robotic wildlife now dominates the landscape. However, in telling this tale, it strives for grandeur, and epicness, and portent, and it merely serves to underline how needlessly bloated video game storytelling has become.


It certainly doesn't help that the voice acting is all over the place. Even within one tribe there's no consistency of performance. You'll get teens who talk like they're hanging around a bus station butting up against wannabe Vikings. The dialogue is tiresome and dull, characters are fundamentally unlikeable and difficult to root for, and the cut-scenes might as well just be a deafening klaxon, and somebody shouting "THIS BIT IS IMPORTANT!" over and over.

And then I played Zelda: Breath of the Wild, and got the answer to a question that I didn't even know I wanted answering.
SURPRISING ADVENTURE
What's surprising about Breath of the Wild is just how much it has been influenced by games of recent years. What is really lovely about that is how Nintendo has cherry-picked the parts of the open world genre that it likes, but ignored most of it. Unlike the spineless approach of most developers, they had the guts to trust that less is almost always more.

It's like they created a game specifically for me. Everything I've raged about in recent years, about tedious cut-scenes, bad acting, awful characterisation... I can't criticise any of that in Breath of the Wild, because it simply isn't there.

Your main character is a blank slate. There's no bad acting, because everything is conveyed through text. The cut-scenes are kept to a minimum, and when they do appear they're utterly charming - because they're not pretending to be movie clips. There's something wonderfully life-affirming and optimistic about everything that Breath of the Wild does.

However, perhaps its greatest accomplishment is in how it has humiliated the entire video games industry. It has shown them exactly where they've been messing up in trying to emulate cinema. It has shown them that their bloated storytelling is thoroughly unnecessary.

​If you follow the story in Breath of the Wild - and you don't even have to do that, if you don't want to - you'll be treated to a tale which is all the more epic and spine-tingling and magical for the economical way in which it's being told.

Indeed, even when the story isn't being told, it still surrounds you. The sense of history in Hyrule is palpable - and despite taking place in a cartoonish, cel-shaded world, is far more evocative and convincing and consistent than Horizon Zero Dawn's misjudged, tonally wonky, hipster-populated, future.
WOKE
I'd love for Breath of the Wild to be a wake-up call to the games industry, but I'm not optimistic. While, in my opinion, Nintendo has shown how video games should always handle their storytelling - basically, how it used to be done - most developers seem addicted to their bloat. 

Instead of asking themselves how video games can tell a story - through gameplay, through art design - there seems to be an endless quest to top the previous most-epic game. There's something depressing and spineless and unimaginative about it. It's the opposite of true creativity.

Apart from certain independent titles, and the games of Naughty Dog and - perhaps - Rockstar, I've not played a game in years which has been able to tell a story that could compete with the best cinema has to offer. Clumsy, leaden writing, and an utter inability to make the player feel for the characters, has now become the norm. Do we enjoy it, or have we merely learned to tolerate it?

The problem seems to stem from an obsession with trying to prove that games can stand shoulder to shoulder with cinema. Except... they hardly ever do. And the infuriating thing is that hardly anybody stops to ask whether trying is even the right thing to do.

By stripping away the sort of graphical clout of something like Horizon Zero Dawn, by reducing the narrative to its bare bones, Breath of the Wild is more emotional, more effective, more engaging, more epic than The Witcher III, Dragon Age Inquisition, Skyrim, and pretty much any other RPG you can think of. Not to mention the Call of Duty series, or... well, basically any game which forces us to suffer through a cut-scene. 

Too often, video games are trying to be something else. Or - at the very least - feeding off one another, and not stopping to ask whether the games they're plagiarising from are getting it right. 

Unfortunately for them, Breath of the Wild has just proved that they're all doing it completely wrong. It revels in being a video game, and the potential that offers. It makes abundantly clear that video games were getting storytelling right decades ago, before you all ruined it.

​Thank Mumm-raa that Nintendo still realises it, and Gawd bless their integrity and stubbornness.
Read the whole story
psyq
2827 days ago
reply
I'm very happy Nintendo has not forgotten how to create real games that feel like games. As opposed to non-games that are half-assed movies.
Switzerland
Share this story
Delete

Selbstfahrende Autos: US-Regierung drückt auf die Tube

1 Comment
Anthony Foxx

Plötzlich geht es schnell: In den letzten Monaten seiner Amtszeit will das Team um Barack Obama das Fundament für die Einführung autonomer Fahrzeuge legen. Das soll Leben retten. Unausgesprochen bleibt das Ziel, Asien und Europa zu überrunden.

Read the whole story
psyq
2996 days ago
reply
So the race for autonomous cars becomes in 2016 what the arms race and the space race were earlier.
Switzerland
Share this story
Delete

'What this video game needs is actual footage of real gruesome deaths'

1 Comment

Animal rights group wants input into design of Farming Simulator 17

Protests about video games usually call for less violence. But People for the Ethical Treatment of Animals (PETA) has called for more violence - and more graphic violence – in the forthcoming Farming Simulator 17.…

Read the whole story
psyq
3003 days ago
reply
Why not? While they're trying to be realistic, they might as well show the reality of gruesome animal deaths.
Switzerland
Share this story
Delete

saint

1 Comment and 5 Shares

saint

This month’s raffle winner is Alan from Wisconsin. Congrats, Alan – you win a signed print of your choice.

Have you considered becoming a Patron of Jesus & Mo? Click the image below to see what it’s about. Rewards abound!


.

Read the whole story
psyq
3005 days ago
reply
Switzerland
Share this story
Delete
1 public comment
ScottInPDX
3006 days ago
reply
Seems the bar for miracles got lower after God died...
Portland, Oregon, USA, Earth
Next Page of Stories