How to use a proxy with LWP::UserAgent in Perl?

Using a proxy with LWP::UserAgent in Perl is straightforward and flexible. LWP::UserAgent allows you to configure HTTP, HTTPS, and FTP proxies by setting the proxy URL for the relevant protocol. This is especially useful when you're behind a corporate firewall or want to route requests through a specific proxy server.

Setting Proxy in LWP::UserAgent

To configure a proxy, use the proxy method on your LWP::UserAgent object. It accepts two parameters:

• PROTOCOL — usually http, https, or ftp.
• PROXY-URL — the URL of the proxy server, e.g. http://proxy.example.com:8080/.

You can set proxies for multiple protocols individually if needed.

Example: Using a Proxy with LWP::UserAgent

use strict;
use warnings;
use LWP::UserAgent;

# Create a new user agent
my $ua = LWP::UserAgent->new;

# Set proxy for HTTP and HTTPS requests
$ua->proxy('http',  'http://proxy.example.com:3128/');
$ua->proxy('https', 'http://proxy.example.com:3128/');

# Make a simple GET request to demonstrate proxy usage
my $response = $ua->get('http://httpbin.org/ip');

if ($response->is_success) {
    print "Response received through proxy:\n";
    print $response->decoded_content, "\n";
} else {
    warn "Failed to fetch URL: ", $response->status_line, "\n";
}

In this example:

• LWP::UserAgent->new creates an HTTP client.
• $ua->proxy('http', 'http://proxy.example.com:3128/') tells the user agent to route all HTTP requests through the specified proxy server.
• A simple GET request is done to http://httpbin.org/ip to show the IP address as seen by the target server (demonstrating proxy usage).
• is_success checks if the HTTP request succeeded.

Perl Concepts: Sigils and TMTOWTDI

The $ua variable uses the scalar sigil $, because LWP::UserAgent->new returns a single object instance (a scalar reference). Methods like proxy and get are called on this object.

Perl’s philosophy “There's More Than One Way To Do It” (TMTOWTDI) means you can also set proxies via environment variables http_proxy or https_proxy, which LWP::UserAgent respects by default. But setting proxy via the proxy method allows per-instance customization.

Common Pitfalls

• Missing trailing slash in proxy URL: Always include a trailing slash in the proxy URL (e.g., http://proxy.example.com:3128/) to avoid confusing LWP.
• Proxy authentication: If the proxy requires authentication, you can include it in the URL, e.g. http://user:pass@proxy.example.com:3128/.
• HTTPS proxy: Note that HTTPS proxy support in LWP::UserAgent uses the HTTP proxy protocol; you cannot specify a proxy with https:// scheme.
• Environment variables: If proxies are set via environment variables, explicit proxy calls override them.

Version Notes

The proxy method and general behavior have remained consistent in Perl 5 versions well before 5.10. The example works with all modern Perl 5 releases.

In summary, configure your proxy by calling proxy on your LWP::UserAgent object, specifying protocol and proxy URL. This integrates smoothly with existing Perl HTTP scripts and gives you fine control over your HTTP traffic routing.