Using rules engine to correctly account for leap years

I currently have a rule built out that looks at a certain date field (we'll call it Date XYZ) and when the current date (rule date) is 365 days ahead of "Date XYZ", it triggers the setup action. Within the Setup Rule, the only way to determine if a certain date is exactly 1 year in the past is through the "Subtract N Days from Rule Date". This works fine with N = 365 except for leap years where N needs to equal 366. Doing a simple OR statement won't give me the correct results for when the year is a non-leap year. Anyone have any experience working with year anniversary based setup actions accounting for leap years?