Well for starters....it may well be illegal to be that close to the tracks. It depends on the state and the railroad. Check first.

If it is legal....if you are talking about riding on the ballast stones (the 1 inch sized rocks that go between the cross ties to keep the tracks from sliding around) you can't ride on that stuff. It's too loose. You'll wear yourself out fast. Some railroads have ballast stone that only extends a foot from the side of the cross ties. Some have ballast beds wide enough for a whole extra track to rest on.

If you are talking about the hard pack access paths a few feet off the tracks, it'll be a great ride. Low grade elevation for sure. Probably relatively smooth. If it's hard pack limestone...wonderful.

This also makes a difference in legality. Riding on the tracks or within 3 feet of them is almost certainly illegal. (Not to mention really stupid, depending on the type of train. Heavy freight trains don't exactly sneak up on you so you'd have plenty of warning with all the noise and ground rumbling. But light rail commuter trains can get up on your fast and relatively quiet) Riding on the access roads next to the tracks depends on where you are and what railroad it is.