What it mostly is is a day off from work for office workers. (On the other hand it's a day of MORE work for retail, restaurant and other service industry folks.)
So why is there this pressure surrounding these types of 3-day weekends? Why do I always feel vaguely defensive about NOT planning to grill things and drink copious amounts of beer and hang around in a swimsuit all day? Why do I feel I let people down when they ask, "Got any big plans for Labor Day?" and I sheepishly reply, "Well...I guess we'll watch some tv, maybe do a couple of loads of laundry, maybe go out for some tacos later on." Clearly, based on the response I get from this reply, this is not the sort of big Labor Day plans they expected to hear about.
Fortunately the Dubious Holidays are over for a nice long spell. (But don't even get me started about Valentine's Day, the "holiday" where no matter WHAT you do for your loved one, you feel like a failure...)