Lab 3: Web Scraping Protection¶

Connect to the lab environment¶

1. From the jumphost, launch chrome, click the BIG-IP bookmark and login to TMUI. admin/password
2. From the jumphost, launch firefox, which we will use to create the macro.

Remove any existing security policy from the Webgoat Virtual Server¶

1. On the BIG-IP TMUI, Go to Local Traffic > Virtual Servers > asm_vs
2. Click the Security > Policies tab at the top
3. Change the Application Security Policy to “Disabled”
4. The Logging Profile should be set to “Log Illegal Requests” and click update

Connect to the Webgoat Application¶

1. Using Firefox, click on the shortcut for WEBGOAT login

   http://10.1.10.145/WebGoat/login

Create a web scraping macro¶

1. Launch the iMacros sidebar by clicking on the icon at the top-right of Firefox

2. Click the iMacro Rec menu, then click the Record button
3. On the pop-up that asks to close all tabs, select No
4. Click Stop to save the current macro (URI should be /Webgoat/login )
5. Click the Play menu and set the Max to 12 and click Play Loop
6. Did the pages load successfully?

Create a security policy to prevent webscraping¶

1. Log into the BigIP through the browser
2. Click on Security > Application Security > Security Policies and Create
3. Select the Advanced view instead of Basic (default)
4. Name the policy “webscraping”
5. Select “Rapid Deployment Policy” for the “Policy Template”, this will bring up a prompt asking if you want to continue, click “Ok”
6. Select “asm_vs” for Virtual Server and click Create Policy (upper left)
7. Change Enforcement Mode to “Blocking”
8. Once created, go to Application Security > Anomaly Detection > Web Scraping
9. Click Bot Detection and select “Alarm and Block”. This will bring up a “Bot Detection” menu below
10. Edit the settings per the screenshot, click Save and then Apply Policy

Create a DNS Resolver¶

1. You can either follow the link in the warning as you enable Web Scraping, or go to Network > DNS Resolvers > DNS Resolver List and Create
2. Assign a name to the Resolver profile and click Finished

Attempt to scrape the Webgoat Login Page¶

1. Go back to your Webgoat tab in Firefox and re-run the macro you created
2. Did the page hits load successfully?

Review the Security Event Logs¶

1. Go to Security > Event Logs > Application > Requests
2. You should see some current illegal requests, as in the example below, click on one and examine the details

3. What caused ASM to block the request?
4. Now go to Security > Event Logs > Application > Web Scraping Statistics
5. Do you see any events?

Reset the Virtual Server config for the next lab¶

1. Clear the app security event log by going to Secuirty > Application Security -> Event Logs > Requests and clicking the check box to select all “Illegal Requests”. Then click “Delete Requests”.
2. Remove the webscraping security profile from the asm_vs virtual server by going to Local Traffic > Virtual Servers > asm_vs, then click Security > Policies tab. Then set “Application Security Policy” to Disabled and click Update.