43-44% of websites are Wordpress. Many non WP sites still have public APIs. Besides the legality of ignoring the robots.txt, it's also just the kind and courteous thing to do.